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ABSTRACT 
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J,000 separate collections had been distributed to American 
educators. Regarding quality control, both reviewer-based and 
"selection-assistance” strategies were used during the project to 
improve the quality of objectives. In particular, the 

situation-assistance strategy involving ratings of extant objectives 
by various clienteles, e.g«, students, parents, and teachers, was 
extensively field tested. For test items, both a priori and a 
posteriori procedures were employed with certain advantages noted for 
each. An extensive analysis of competing classification schemes 
failed to yield any markedly innovative category systems which 
appeared particularly useful. Nonetheless, recommendations were 
reached for revised classifications of objectives in the various 
collections, chiefly on the basis of more limited numbers of content 
general objectives. (Author) 
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AUTHOR'S ABSTRACT 



This report describes a two— year project designed (1) to de- 
velop collections of measurable instructional objectives and 
related assessment procedures, (2) to devise methods for im- 
proving quality control of such materials, and (3) to con- 
sider alternative schemes for categorizing these objectives 
and related measures. As a consequence of rhe project, 28 
separate collections of instructional objectives and related 
test measures were produced, in all, over 2,000 objectives 
and more than 7,000 test items to measure these objectives. 
During the period of the project's existence, well over 
30,000 separate collections had been distributed to American 
educators. Regarding quality control, both reviewer-based 
and ^ selection— assistance" strategies were used during the 
project to improve the quality of objectives. In particular, 
the situation-assistance strategy involving ratings of extant 
objectives by various clienteles, e.g., students, parents, 
teachers, was extensively field tested. For test items, 
both ji priori anc* ^ posteriori procedures were employed with 
certain advantages noted for each. An extensive analysis of 
competing classification schemes failed to yield any markedly 
innovative category systems which appeared particularly use- 
■*■'^1 * Nonetheless , recommenda.tions were reached for revised 
classifications of objectives in the various collections, 
chiefly on the basis of more limited numbers of content gen- 
eral objectives. 
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CHAPTER ONE 



PROJECT OVERVIEW 



This report describes a two-year project, supported by the 
United States Office of Education, which was designed to 
develop sets of measurable instructional objectives and re- 
lated assessment devices for use by American educators. The 
project was initiated at a time when, although there was 
widespread advocacy of the proposition that educators should 
employ measurable objectives in their instruction and evalua- 
tion efforts, few comprehensive collections of such objec- 
tives were available. The project was closely coordinated 
with the activities of the Instructional Objectives Exchange, 
a project initiated by the UCLA Center for the Study of Eval- 
uation to serve as -a developer and disseminator of objectives 
and related test measures. By increasing the magnitude of 
objectives available to educators through such agencies as 
the Instructional Objectives Exchange, this project was de- 
signed to facilitate the efforts of those school people who 
wish to design instruction and assess the quality of their 
designs in terms of measurable learner outcomes. 

The impetus for the project occurred in early 1969 when the 
principal investigator was contacted by Office of Education 
officials to explore the possibility of undertaking a project 
to develop collections of objectives and related test measures 
from which educators might select, particularly such educa- 
tors as those associated with the network of schools (ES "70) 
which USOE was at that time fostering. Administrators of this 
group of schools, and they appeared representative of a num- 
ber of such innovatively inclined educators , were essentially 
pleading for the creation of well organized pools of instruc- 
tional objectives and measures from which their staffs could 
choose, rather than being obliged to construct such objectives 
and measures themselves. The principal investigator parti- 
cipated in a meeting in Washington, D.C., where representatives 
of a number of these schools met with USOE officials to in- 
dicate the nature of their plans with respect to the evalua- 
tion of their innovative efforts. When the necessity for 
measurable instructional objectives arose in connection with 

evaluation activities , these representatives were inter- 
ested in learning of the establishment of the Instructional 
Objectives Exchange (IOX) which only a few months earlier 
had commenced operation at UCLA. A description of the objec- 
then being generated within IOX revealed, however, that 
most of the materials were aimed at the elementary grades 
whereas this particular group of innovative schools was con- 
cerned almost exclusively with education at- the secondary 
level. Accordingly, they urged chat additional support be 
given to the Objectives Exchange to hasten its expanded dev- 
elopment of objectives and measures at the secondary level 
and, at least for the ES '70 schools, in vocational fields. 
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Accordingly, after additional discussions with Office of Edu- 
cation officials, a proposal to support a two-year develop- 
mental effort was submitted to the Office of Education in 
Apni, i969, and the project was approved several months 
thereafter, commencing formal operation on July 1, 1969. 



While its overall goal was to increase the number of objectives 
and measures available to American school people, there were 
to be three specific purposes of the project. Because these 
three purposes served both a s the focus of project activities, 
as weli as the outline for the remainder of this report, they 
will be briefly identified at this point. y 



Purpose Number _1. The initial purpose of the project was 
to increase the magnitude of available objectives and meas- 
ly 3 available to American educators. While it was anticipa- 
ted at the time the grant was awarded that these materials 
would be distributed through the Instructional Objectives Ex- 
change , their availability in the public domain would also 
permit them to be distributed through other comparable item 
bank agencies. 



rur pose _ Number 2 . The second purpose of the project was 
to exercise improved quality control over acquired and devel- 

°?® d ??- ,e S t i V S S ' 4 .^ Both V ith respect to extant objectives which 
were located by the project staff as well as those which were 
developed by the staff , it was considered particularly import- 
ant to impose high quality standards on the materials. Pro- 

o^ C Tti S +. SU< ^ h n aS ? an ' of course ' produce an immense quantity 

°| n ? aterlal ® whlch 1S ? f such poor quality that the materials 
will never be worth using. The second purpose of the pro- 

devilooL t0 d” Pr ? V h the < I ualit y of the objectives and measures 
deveioped and, at the same time, to study the procedures 

whereby such quality control standards could be implemented. 

j_ n on ”vS° Se jl p ber 3 - The third purpose of the project was 
to consider alternative classification schemes which might 
be empioyed to both develop and retrieve instructional objec 

torJ S COuld be more readily employed by educa- 

‘ ? pe ^ f lcalby ' an effort was to be undertaken to consider 
i t;L ^ n f che3 ? es °ther than those involving the commonly 
employed grade level and subject rubrics. ^ 

^ 0 ?i d ^ Clear ^ r ° m an exarnin ation of the above three pur- 
P this pro jec . can be more accurately described as 

a de velopment effort rather than a classic research effort. 

WaS ?® sl 5 ned to produce sets of high quality 
W ^ 1Ch V uld serve as useful resource for American 
b 1 P eople * T here were no focal hypotheses to be tested 
no elaborate statistical analyses of resulting data. Rather, 

? h ?ct P H 0:,eC 5 re P resent ed an intense effort to produce mater- 
WmL' n ® cesaa fY for certain educational activities. 

While the tools of the conventional educational researcher 
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were employed sporadically to test the viability of procedural 
variants for certain developmental activities, the thrust of 
the enterprise was clearly developmental. Thus a development 
paradigm rather than a standard research model was employed. 



Organization of the Report 

This final report is organized around six major chapters , the 
first of which is designed to provide an overview of the en- 
tire project. In the current section, for example, identifi- 
cation of the remaining chapters will be presented. Follow- 
ing the identification of these chapters , a brief historical 
account of the project's relationship with the UCLA Center 
for the Study of Evaluation will be given as well as its 
interaction with the Instructional Objectives Exchange when 
that agency became separated from the Center. The current 
chapter will be concluded with an introductory examination 
of the three separate purposes of the project and will offer 
what is, in essence, a preview of the remaining five chapters 
of the report . 

The five chapters to follow the current Overview Chapter are 
the following: Chapter Two presents a rationale for the es- 

tablishment of objectives bank operations comparable to the 
Instructional Objectives Exchange. Since the goal of the cur- 
rent project was to prepare materials which might be employed 
by such agencies, it was considered desirable to depict why 
such agencies may be needed. Chapter Three describes the 
early, middle, and later developmental activities conducted 
within the project and attempts to detail an increasingly self- 
conscious developmental enterprise. Chapter Four describes 
the principal product of the project, namely, the collections 
of instructional objectives and test measures produced during 
its operation. In that chapter collection, titles, number of 
objectives, number of test items, and general content for 
each objective collection will be presented along with informa- 
tion regarding dissemination of materials to date. Chapter 
Five treats the quality control activities associated with 
the project , offering a general description of the varied 
techniques which were employed to assure high quality of the 
objectives and related test measures. Chapter Six deals with' 
the activities regarding the classification of objectives and 
will present recommendations resulting from two years ' consid- 
eration of the types of classification systems which might 
prove optimally useful to educators . 



Project's Relationship with CSE and IPX 

It was noted earlier that the current project was initiated 
at a time when the Instructional Objectives Exchange had been 
recently created as a project within the UCLA Center for the 
Study of Evaluation (CSE) . In all of the language of the 
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original grant proposal, the nature of this anticipated re- 
lationship was clearly spelled out. Specifically, there would 
be a collaboration effort involving resources of the Center 
for the Study of Evaluation and the new project staff. In- 
deed, this was precisely what occurred, for upon notification 
that the grant had been approved, personnel from CSE and the 
current project worked in close cooperation for the entire 
first year of the project's existence. Several key CSE sup- 
ervisory personnel cooperated with the project's principal 
investigator to coordinate our work force, most of whom were 
housed within office space provided by CSE. By capitalizing 
on certain equipment and support facilities of CSE, it is 
clear the project was able to accomplish more during this 
fl— st year of operation than would have been the case without 
*^SE interaction. At the same time, however, information being 
derived from the project was being used by a more comprehensive 
research effort of the Center , the Project for Research on 
Objective Based Evaluation, which was directed by a guiding 
committee constituted by a number of UCLA faculty including 
the principal investigator of the current project. In other 
words, the symbiotic relationship between the project and CSE 
was deemed exemplary. In many ways, it was unfortunate that 
this relationship could not continue during the second year 
of the project's existence. 

With the approach of the second year, however, USOE officials 
guiding the operations of the Center for the Study of Evalua- 
tion had indicated their desire to CSE leaders that the In- 
structional Objectives Exchange project be separated from the 
Center. It was the view of these USOE officials that the 
IOX represent a service rather than a research and develop- 
ment operation and, therefore, should more appropriately func- 
tion as an agency separate from the research and development 
oriented center. As a consequence, in June of 1970 the In- 
structional Objectives Exchange was established as a separate 
non-profit corporation in the State of California and moved 
to quarters some distance from the University. Because office 
space was at a premium, the current project was obliged to 
move from the Center and, more decisively, from the campus 
as well. An off-campus suite of offices located near the 
University served as the base of operations for the project 
during its remaining year. During that period,, however, 
reasonably close relationships were maintained between a num- 
ber of key staff within the Center for the Study of Evalua- " 
tion and the Instructional Objectives Exchange, for it was 
still anticipated that the Instructional Objectives Exchange 
would serve as a primary outlet for materials being developed 
by the project. Although all such materials were destined 
for the public domain, where they could be used by any objec- 
tives bank agency wishing to utilize them, it was assumed 
likely that the Instructional Objectives Exchange would be one 
such agency distributing the materials. 
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Purpose Number 1: Developing Objectives and Items 

The principal activity during the two-year period was, con- 
sonant with the chief thrust of the project, to develop new 
collections of instructional objectives and test measures. In 
all, 28 such collections were developed. The total number 
of objectives produced exceeded 2,000. Finally, over 7,000 
test items related to these objectives were prepared. During 
the early months of the project, some of these materials were 
gathered from existing sources throughout the nation. Exten- 
sive efforts were undertaken to solicit educators to contribute 
objectives and items for -che project., chiefly through the In- 
structional Objectives Exchange. Certain of these incoming 
collections were judged suitable as points of departure; thus 
our staff was able to build on sue i materials in the develop- 
men t- of new collections. As tie rro ths progressed, a~i in— 
C3 f ea ?ingly 9 re 9t er portion of c elc pment activity occurred 
within the project itself , with :ewf.r and fewer materials being 
ised from the outside. During the final months of tht project, 
almost all dev -lopment consisted ef producing completely new 
objectives and assessment devices. As the project proceeded, 
the staff also became increasingly aware of the steps associa- 
ted with our development enterprise. Accordingly, we began 

involve more and more carefully defined techniques for pro- 
ducing new objectives and test measures. With the most recent 
collections, for example, far better delineated check points 
were utilized and more test measures per objective were pro- 
duced. To illustrate, with some of the earlier collections 
only one test item per objective was developed, whereas with 
almost all subsequent collections at least six items per ob- 
jective were produced. 

The response by American educators to the materials we have 
developed has, thus far, been extremely positive. To illus- 
trate, during the two-year period of the project's existence, 
w®ll over 30,000 separate collections have been distributed 
to American educators. These materials, at least those being 
distributed by the Instructional Objectives Exchange, were 
being sold (with all revenues being used for subsequent dev- 
elopment and revision of the materials). Educators who ac- 
the objectives generally had to purchase them, and 
thus a great many educators backed up their interest in such 
materials with actual financial expenditures . 



Purpose Number 2: Quality Control 

One of the more interesting areas of activity in connection 
with the project was related to our efforts to improve the 
quality of the materials we were developing. in the early 
days of the IOX operations, the essential mode of operation 
was "rough and dirty” with the beginning collections resem- 
bling materials which had been raked together rather than 
systematically developed. With the availability of funds 
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from the current project, however , we were able to be more 
systematic in appraising the materials and particularly were 
able to take advantage of a nationwide network of schools 
created by the Center for the Study of Evaluation. This 
national network of approximately a dozen schools supplied 
us with ample opportunities to verify the quality of our 
materials prior to release. For example, we were able to send 
first draft and second draft collections to teachers and ad- 
ministrators within these _chools for review. As the project 
continued, these reviews became increasing: -stematic and 

we were able to initiate a number of check-c. . :ints re- 
garding external reviewer approval prior to c . s cri outi on. 

We also began to explore the manner in which : bjecoivs - 
should be revised, particularly those in the aife* tiv^ domain. 
Certain collections of objectives developed by the Inr;ruc- 
tional Objectives Exchange under funds from ot:.a: juar ers 

were systematically scrutinized on the basis of eld _ast 
information. For example, construct validation e proa.hes 
were utilized in an effort to judge the validity £ ce ;ain 
affective measures regarding children's (1) attit "les toward 
school and (2) self concepts. 

As the project progressed, we began to view qual - zv control 
in two different ways. First, there was the question cf the 
formal quality of the materials , namely , were ws producing 
information which was on the face of it inaccurate, im- 
precise, etc. The second view of quality control related to 
the value which different individuals placed on the materials. 
Were certain objectives viewed as more important than others? 
^i*i given groups view particular objectives as more praise- 
worthy than others? Employing this scheme, we were able to 
investigate the efficacy of securing preference rating informa— 
f rorn various quarters and, by making such information 
available, provide useful data for those who must choose 
among objectives. In a sense, thereby, these objective ".el- 
ectors were able to exercise their own quality control on 
the basis of increased access to information which would aid 
them in making choices among competing objectives. During 
the middle and latter phases of the project, we were able to 
try out variations of these schemes for gathering preference 
data and have reached the conclusion that the procedure repre- 
sents a viable scheme for guiding educators as they employ 
their own control mechanisms. 

We have also looked carefully at alternative methods of estab- 
lishing high quality for our test measures. Both a. prior i 

£* posteriori approaches were employed during the project 
to establish the congruency between objectives and measures. 

The use of Hively ' s item form approach as well as a number of 
empirically based tactics were investigated. These activities 
are described in more detail in Chapter Five of the report. 
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Purpose Number 3: Classification Activities 



Of the three purposes of the project, the classification task 
was by far the most perplexing. From the very beginning, we 
attempted to capitalize on previous efforts of those concerned 
with such classification enterprises and discovered that very 
few classification procedures had been developed which would 
prove of any utility. During the early months of the projec 
research assistants scoured the literature in an effort to 
find productive alternatives to typical grade and subject 1 * 7 el 
classification schemes. When we discovered that the litere 
ture was deficient with respect to such categorization scheme 5 , 
we turned our attention to securing input from some of the me 3 t 
able educational thinkers in oxir country. We were able to tap 
a half dozen or so of the very best minds in the field re- 
garding how this task might be consumated. In gerneral, 
both internal efforts from the staff and suggestions from the 
outside proved less than satisfying. The classification task 
has, indeed, represented a frustrating arena. We have, how- 
ever, set down our best recommendations regarding the manner 
in which objectives might be classified and have examined 
certain procedural questions associated with such classifica- 
tion efforts. These deliberations and resulting recommenda- 
tions are described in Chapter Six. 



Summary 

In review, then, this two-year project represented an effort 
to increase the number of objectives and related test measures 
available to American educators, to improve the quality of 
such measures (and to study the manner in which such quality 
could be assured) , and to consider alternative ways to clas- 
sify these materials for subsequent retrieval. In the five 
chapters to follow, it is hoped that activities of the pro- 
ject related to each of these three purpeses will be described 
in sufficient detail to prove of value to researchers and 
developers engaged in comparable pursuits. 
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CHAPTER 2 



A RATIONALE FOR 
OBJECTIVES -BANK OPERATIONS 



Because this project was conducted throughout in coor- 
dination with the activities of +*he Instructional Objectives 
Exchange, both when IOX was a project of the UCLA Center for 
the Study of Evaluation and, more recently, when IOX had 
operated as a nonprofit educational corporation, it is impor- 
tant to understand the rationale underlying the creation of 
the Instructional Objectives Exchange. The key ingredient in 
a useful system which can provide objectives and measures to 
American educators consists of the basic materials available 
through that system. This project, of course, was designed 
to make available more objectives and measures which could be 
used in such systems as that provided by the Objectives Ex- 
change . 

This chapter, therefore, will describe the basic reasons 
why objective-bank operations are considered useful assests 
for outcomes-oriented educators. It is gratifying to note 
that in recent months other agencies* have begun to make 
available collections of objectives and measures. Undoubtedly 
such agencies will be making use of the public domain materials 
produced as a consequence of -this project. We can turn, there- 
fore, to an examination of the rational underlying the Instruc- 
Objectives Exchange, insofar as it is representative of the 
objectives— bank operations which are beginning to emerge through- 
out the nation. 



Rationale 



"The quality of any instructional sequence must be evalu- 
ated primarily in terms of its ability to promote desirable 
changes in the intended learner." This assertion, or statements 
similar to it, have met with increasing approbation among influ- 
ential American educators during recent years. Not that it 
represents a novel conception 7 one could undoubtedly locate 
comparable utterances from the very beginnings of educational 
history. But the increasingly widespread agreement with this 

* The Directory of Measurable Objectives Sources can be obtained 
from the Upper Midwest Regional Educational Laboratory, in 
Minneapolis, Minnesota or in care of Mr. Arthur Olson, Colorado 
State Department of Education, State Office Buidling, Denver, 
Colorado 80203. Objectives and related tests of the Wisconsin 
Design for Reading Skill Development, an individualized reading 
system, are also available from National Computer Systems, 4401 
West 76th St., Minneapolis, Minnesota 55435. 
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conception of .structional effectiveness is new. Never before 
a fhis counts. have we had so many educators manifesting com- 
mitment to the notion that we should judge instruction primarily 
by the results it produces in learners. 



Criterion - Referenced Instruction 

Perhaps the type of instructional strategy beinc. advocated 
these days can best be described as criterion-referenced instrvc 
_|ion. This approach to instruction focuses primarily oln the 
degree to which the learner can perform specified criterion 
behaviors. For example, in preparing instructional materials 
fcne developers decide wh.a.t: to revise on the basis of learner 
performance data, not according to the judgment of consulting 
experts. Or in another situation, a school district decides to 
select one set of supplementary reading texts, instead of another 
-.ecause of pupil performance on related criterion tests, not 
because one set of texts is more attractively illustrated than 
the ether. Such examples accurately suggest that a primary 

criterion— referenced instruction is a preoccupation 
Vifcn the results of instruction, not the procedures used to 
promote them. It reflects an ends-oriented approach to instruc- 
tion r dther than means-orienced approach. Since most educators 
concur that the ultimate index of an educational program's worth 
is the degree to which it benefits the learner, the increased 
support of criterion-referenced instructional approaches is 
gratifying. 



But against the increasingly supportive backdroo. it is 
distressing that very few large-scale criterion-referenced 
instructional operations are underway. Verbal support is there, 
widespread practical implementation there is not. Why? 

A Time - Consuming Task . The principal deterrent to expanding 
the extent of criterion-referenced approaches used in the nation's 
schools is fairly easy to identify. Developing criterion mea- 
sures of sufficient quality and satisfactory breadth is too much 
worx for most educators. Developments regarding the use of 
here V1 ° r ^' 1 ^ stated educational objectives may be instructive 



Much of the recent agitation regarding the desirability of 
describing instructional objectives in terms of measurable 
® arne3r behavior is based on the belief that operationally 
seated objectives will mors readily permit educators to assess 
the impact of instruction where it should be assessed, namely, 
in modified learner behavior. But many proponents of operation- 
a il Y st f ted educational objectives are beginning to complain 
about the paucity of such objectives in the schools. Educators 
c^n be informed of the merits of behaviorally stated objectives; 
they can be taught to state objectives properly; they can even 
become quite enthusiastic about the desirability of stating 
objectives behaviorally. But few of them do it. The reason is 
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not unwillingness but, instead, reflects a lack of wherewithall. 
Teachers are already too burdened to find the time to develop 
operationally stated objectives for their classes. School 
districts have already committed their increasingly limiued 
resources to other tasks. In those isolated instances where 
there has been an effort to develop precise instructional 
objectives on a large scale, the participating educators will 
readily admit how taxing the enterprise has been. 

Imminent Duplication . The financial and personnel costs 
of the isolated projects to develop instructional objectives 
points up another problem. In spite of the difficulties asso- 
ciated with the development of explicit objectives, some dis- 
tricts are undertaking the task. For example, several years 
ago the Clark County, Nevada School District developed a set 
of behaviorally stated oojectives for mathematics instruction, 
grades K through 6. There are other examples of such endeavors 
in various parts of the U.S. 

The absence of any scheme through which one district could 
become aware of the existance of similar developmental projects 
makes it probable that a distressing amount of duplication w? 11 
occur among those few educators who are zealous enough to attempt 
the development of precise instructional aims. For instance, 
more than a year after the Clark County, Nevada schools had com- 
pleted their preparation of K— 6 instructional objectives for 
mathematics, two districts in different states commenced work 
on precisely the same project. They were unaware of the Clark 
County objectives. The wheel was about to be re— invented. 

Not that the Clark County objectives would satisfy all 
districts; undoubtedly there would be modifications. But the 
energy that could be saved nationally by adapting extant sets 
of objectives rather than starting from scratch is incalcuable. 
For example, several of the USOE-supported regional laboratories 
are investing significant resources in encouraging educators to 
develop operationally stated goals. The probable overlap be- 
tween such efforts and similar projects initiated by local dis- 
tricts is considerable. 



Objective— Generators and Ob jective— Selector s 

It has become increasingly clear to those who have been 
promoting the use of operationally stated objectives that it 
may be expecting too much to ask already harassed teachers and 
administrators to generate their own objectives. It is an 
arduous task and, although the teacher may be willing to state 
his objectives behaviorally, under present conditions most 
teachers just can't find the time to do it. But though objec - 
tive-generation may be too demanding, ob jective- select ion should 
not be. If the instructor's task were simply to choose from 
comprehensive sets of operationally stated objectives those 
which he wished to achieve, his task would be manageable. He 
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cou 'f d follow through on his commitments to precisely explicated 
goals without being obliged to construct all such goals himself 
But, obviously, someone needs to construct the objectives from 
which he can select. 



Local Option. Under any scheme in which the educator is 
the selector rather than generator of objectives there may be 
some concern regarding the degree to which the objectives will 
be imposed from above." A viable objectives selection scheme, 
however, should permit just that — the selection of obiect-i . 
If particular objectives are not preferred, they are not selected. 

°£ the objectives are not available which the selector 
favors, he can always generate additions. Having selected the 
bulk Oi his goals from those prepared by others, such an objec- 
tives generation task should be manageable. Local autonomy in 
the selection of objectives should be an integral part of any 
objectives selection scheme. The availability of objectives 
rom which to choose should increase the educator 1 s range of 
alternatives, never decrease his self-direction. 



Obie ctives Plus Criterion Measures 

Another factor which has not been perceived by all advo- 
ca es o precise objectives is that they be necessary, but by 
themselves they are far from sufficient. Too often even a 

^S‘ a H^ 0 r f lly *.® tated ob j ective ma Y b e used as window dressing 
h^fr^? St £ UCtl ? n aa usua1 *" A Precise objective can be most 
helpfui when planning an instructional sequence, since there is 
clarity regarding the intended post-instruction competencies of 
the learner. But an explicit objective becomes even more useful 

olfSwj f valuate a ? instructional sequence. This can be accom- 
plished by ascertaining the degree to which the objective has 

adhieved. T °.P? rform latter function we need measuring 

devices based explicitly on the objective. A criterion-ref er- 9 
enced approach to instruction requires criterion measures. 

j_ districts have made this logical jump from the develop- 
ed ^ nece ? si ty of developing test items. 

And test items here is used in the broadest possible sense, 

example, including observation of learner behaviors refleC“ 
ing a host of cognitive as well as non-cognitive outcomes. If 
possible for school districts to have access to sets of 
f S g lus “ — - st -l£ ems from which they could choose, then 
selecting certain objectives the district could readily 
SnJJSI de ^ree to which its instructional approaches were 7 

successful. A teacher could evaluate his succXs in achieving 
his goals. The existence of a pool of test items for each 
objective wouid really encourage educators throughout the nation 
to initiate criterion-referenced instructional strategies. 
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The Instructional Objectives Exchange 



Therefore, to encourage increasing numbers of educators to 
adopt criterion- referenced instructional strategies and to 
reduce the probable overlap in objective development efforts, 
the UCLA Center for the Study of Evaluation has established the 
Instructional Objectives Exchange which will serve as a national 
depository and development agency for instructional objectives 
and related measurement devices. The Exchange will perform the 
following functions: 



1. It will serve as a clearinghouse through which the 
nation's schools can exchange instructional objectives, 
thereby capitalizing on the developmental activities 

of other educators rather than duplicating such efforts. 

2. it will collect and develop measuring techniques suit- 
able for assessing the attainment of the objectives 
available through the Exchange. 

3. It will develop properly formulated instructional 
objectives m important areas where none currently 
exist, that is, fill the gaps not covered by available 
objectives. 



The potential impact of such an Exchange, readily providing 
pools of objectives and test items from which districts can 
select, should not be underestimated. With competent staffing, 
a careful developmental plan, and proper dissemination strategies, 
the Exchange could conceivably alter the nature of instructional 
practice in America. 
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CHAPTER 3 



DEVELOPMENT ACTIVITIES 



This chapter jWill describe the nature of the development 
enterprises associated with the production of the objectives 
and measures related to the initial purpose of the project. 
Three sections will essentially detail three temporal phases 
of the development operation, i.e. , the early, middle, and 
latter months of the two-year project. 



EARLY DEVELOPMENT 

In January , 1969 over 8 , 000 copies of a very brief leaflet 
describing the purposes of IOX had been distributed along 
with a questionnaire which educators were encouraged to re- 
turn indicating whether they had access to measurable in- 
structional objectives. We wished to follow up on those 
questionnaire responses indicating the availability of such 
objectives so that they would be relayed, through the services 
of IOX, to other educators. These leaflets were sent, pri- 
marily by mail, to such agencies as the following: superin- 

tendents of major school systems, state superintendents of 
public instruction, directors of curriculum development pro- 
jects . professional associations, subject matter specialty 
associations, etc. We were attempting to accumulate as many 
such collections of measurable goals as we could. 

The kinds of objectives we have received during the early 
months of the project may be of some interest. As of January 
1, ^.970 approximately 325 collections of objectives had been 
contributed to IOX. Most of them were organized around units 
of entire courses within subject matter fields. The majority 
came from public school districts, although there were many 
donated from such sources as ESEA Title III centers, subject 
matter curriculum projects, and individual teachers. Approx- 
imately ten percent of these collections included objectives 
stated with sufficient precision, i.e., measurability, so that 
they could be readily used by the Exchange, even though all of 
our solicitations had asked only for measurably stated goals. 

one percent of the usable contributions fell within 
either the affective or psychomotor domains. 



As these materials became available to the project staff, at 
that time working closely with personnel from the Center for 
the Study of Evaluation, the primary method of development 
consisted of surveying incoming materials to locate those 
which appeared best stated from a measurability viewpoint, then 
coordinating these with other incoming objectives in similar 
subject fields. These materials were then modified, chiefly 
®l lra iiiate unnecessary duplication and to reduce ambiguity 
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°f tlie objectives. In addition, moire objectives were char- 
acteristically devised to fill what appeared to be gaps in 
the coverage since we were not attempting to be prescriptive, 
but were intending to present a wide array of objectives 
from which one could chooser we were anxious not to leave 
any major omission. After a fairly large pool of objectives 
had been created, at least one sample test item was developed 
for each objective, chiefly to further explicate the objective 
In some instances, several items were produced for each ob- 
jective. 



As we were receiving and processing more objectives , a major 
fear aroso, namely, that we would have collected and developed 
a host of objectives and measures but that those materials 
would not be worth very much. We believed we had to get some 
reactions from potential users prior to wholesale distribu- 
tion and, as a consequence, set up a summer institute attended 
by teachers from various parts of the country. A major func- 
tion of these teachers was to serve as a screening group re- 
garding the objectives which had been processed primarily bv 
graduate students until that point. 



Announcements describing the proposed institute were distribu- 
ted in the spring to school systems throughout the country. 
Those interested in participating were advised to indicate 

sul) D e ct matter and grade level of their greatest experience 
and/or inrerest, and identify their educational and profes— 
S i? na i bacJc g roun d in that subject matter. In this way we were 
able to select participants who demonstrated subject matter 
competence and teaching experience. It was our original in- 
tention to work in only four subject areas: mathematics, 

reading, English literature, and primary physical education. 
However, because certain applicants wished to work in fields 
W S? r \ We h , ad suff icient numbers of objectives and items avail- 
able to make a reaction group feasible, it was decided to en- 
large the number of subject areas to include elementary geo- 
graphy, language arts, music, biology, and nutrition. 

The groups were kept small, and. in two areas, music and bi- 
ology, there was only a single individual involved. For the 
most Part, the groups were comprised of from four to eight 
individual 3 . Each group worked under the direction of a sin- 
gle laader ' although in the smaller groups one individual 
served as the leader of more than one group. Usually the 
± ea der was a research assistant trained in the writing of ob- 
jectives and items, and with some experience in the subject 
matter under discussion. in some instances, a leader was 

chosen in advance from among those who had applied as parti- 
cipants . 

The institute took place from July 7 through August 1, 1969 

and was attended by 41 participants from 11 different states. 

The participants lived in a UCLA dormitory, and worked ap- 
proximately six hours a day. Most of this time was spent in 
small groups where the work consisted of going over all objec- 
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tives in the collections IOX was planning to release in the 
fall- Each such objective was discussed, then discarded, 
modified, or retained in its present form. In a similar 
fashion, each item which had been written to test the objec- 
tives was examined. The participants also wrote new objec- 
tives and items , when it was decided that the collection re- 
quired specific additions. Initial training was provided 
for all participants during the first week of the institute 
and for group leaders during the week prior to the institute. 

In our estimate the workshop was successful in accomplishing 
its goals. The objectives and items gathered from the parti- 
cipants perusal were thoroughly examined and defined. More- 
over, many additional objectives and items were written. And 
finally, we believe the participants themselves gained much 
from the experience. As the close of summer neared, a more 
elaborate brochure describing IOX was readied in the form of 
a catalog. The catalog described in detail the background of 
the Instructional Objectives Exchange, and through an insert, 
indicated what collections of objectives would be available 
during the remaining months of 1969., Approximately 15,000 
of these catalogs were distributed by mail. Almost all Amer- 
ican school districts of any appreciable size received a copy 
of the catalog. Once more, a questionnaire was included 
soliciting contributions of objectives to IOX. The actual 
distribution of IOX collections commenced in the fall of 1969. 



Early Status Data 

On January 1, 1970 data were assembled regarding the types 
of response to the two major mailouts, as well as certain in- 
formation regarding the orders for objective collections re- 
ceived at that point by the Instructional Objectives Exchange. 
First, in Table 1 frequencies and percentages are supplied 
questionnaires received either in response to the 
initial January, 1969 mailing or the summer, 1969 mailing. 

It can be seen that as of January, 1970 more than 2,500 
questionnaires have been received. The percentages for dif- 
ferent types of individuals returning the questionnaire are 
presented, and it can be observed that school administrators 
and supervisors supplied almost 60 percent of the returns. 
Questionnaires were received from all 50 states , with Calif- 
ornia, Minnesota, and Pennsylvania displaying the most activity, 
returning 18, 12, and. 9 percent of the questionnaires res- 
pectively. 
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Table 1. 



Questionnaires returned to IOX during 1969. 



Type of Individual Returning 


Frequency 


Percent 


Federal-State Project Administrator 


251 


10.0 


Principal 


355 


14.1 


School Administrator-Supervisor 


1,471 


58.6 


Teacher 


223 


8.9 


College-University Personnel 


183 


7.3 


Other 


14 


0.6 


Not Given 


13 


0.5 


Total 


2,510 


100.0 



We were interested in any differences in the responses to 
the original leaflet sent in January, 1969 and the catalog 
sent in summer. Of the 2,510 responses, 82 percent occurred 
responds to the leaflet whereas only 18 percent occurred 
in r esponse to the catalog. Different colored questionnaires 
permitted this contrast. Obviously, many of those interested 
in IOX responded to the initial rather than the second docu- 
ment. The breakdown of the types of individuals returning 
the questionnaire remained approximately the same in response 
to either the leaflet or the catalog. There were, however , 
some interesting differences with respect to several items 
on the questionnaire that these individuals returned. 

One of the items cited on the brief questionnaire asked whether 
measurable objectives were available to the respondent. We 
hoped to locate available collections of objectives through 
this item. In response to the leaflets, 48 percent of the 
respondents indicated that objectives were available. In 
response to the catalog, only 28 percent indicated the avail- 
objectives. There are alternative interpretations 
for this 20 percent reduction. First, of course, it is pos- 
sible that we reached the majority of individuals who did 
possess such objectives through the initial mailing. An al- 
ternative interpretation is that the more elaborate catalog 
which contained actual samples of measurable objectives may 
have induced the respondents to be more conservative in their 
response to this question. 

Similarly , the percentage of respondents indicating that 
measures for objectives were available to them dropped from 
36 percent in response to the initial leaflet to 20 percent 
in response to the questionnaire accompanying the catalog. 
Similar interpretations could account for this reduction. For 
both responses to leaflet and catalog over 90 percent of the 
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individuals indicated that they would be willing to con- 
tribute any objectives and items which were available to 
them. Ninety-eight percent of both groups of respondents 
indicated they would be interested in participating in the 
Objectives Exchange by withdrawing objectives and items suit- 
a kle for their particular instructional program. 



Early Orders Received 

Once the catalog went out with its description of the 16 sep- 
a 5 at ® collections of objectives which would be available in 
tne fall, orders began to arrive. As of January 1, 1970, IOX 
had received 854 orders representing all 50 states. Once 
mor ®' California, Minnesota, and Pennsylvania led in the 
number of orders with 37, six and six percent respectively. 

°^ ders in excess of five percent. The maj- 

ority of orders arose from local school districts, with 78 
percent of the orders coming from that source, 11 percent 
s ~ at ; e e ^ucational agencies and the remainder 
from private, federal, and other sources. The distribution 
of orders approximated but was not identical to the array of 
questionn^re returns seen earlier in Table 1 # In Table 2 
^ e rec jy enc Y and percentage of orders are represented ac- 
cording to those same categories for questionnaire responses 
d Jf C ' ri ^ >ed in Table 1. as was anticipated, the bulk of the 
requests centered upon objective collections in the areas of 
££ g and ^hematics. Of the 16 collections available dur- 
ing tne fall, the heaviest demands were in these areas. 



Table 2. Orders Received as of January 1, 1970. 



Type of Individual Ordering Frequency Percent 



Federal-State Project Administrator 
Principal 

School Administrator— Supervisor 
Teacher 

College-University Personnel 

Other 

Not Given 



Total 



3 


0.4 


42 


4.9 


360 


37.5 


56 


6.6 


3 


0.4 


93 


10.9 


337 


39.5 


854 


100.0 
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Problems 



There were several classes of problems associated with early 
development activities . The first of these is related to the 
acquisition and retention of a qualified staff. 

Staffing . The key difficulties associated with the 
staffing of any R & D Center visited themselves upon our pro- 
ject. First, of the approximately 30 research assistants 
(graduate students) who were hired at the earliest stages of 
the project, only about six proved sufficiently able and/or 
< iiiiQ e nt to be retained. Although all of these research as- 
sistants were well acquainted with the specification of 
measurable objectives, working in a development context on 
this particular project did not prove appropriate for all of 
them. 



Training . Because our initial collection o' orkers 
were recruited then retained on a selective basis ve undoubt- 
edly trained these people more through the laying on of hands 
than any kind of replicable training procedures. As a con- 
sequence, we continually found ourselves faced vri *• b. the pros- 
pect of employing new staff but having expended c._ - modest" 
energy in developing any systematic training mater: els to he 
used in preparing those staff members tc work on i e project ► 

Research. With this, as with most developmental pro- 
jects, there is a considerable tempts’ ion to emphasize the 
development aspects of the enterprise and neglect research 
inquiry. Although we attempted to conduct research which 
was correlated with our activities, there was a discernible 
tendency to become preoccupied with the development of res- 
pectable collections of objectives rather than carefully 
studying the processes by which such collections are acquired 
and used. 

Distributi on Pressure . During the early days of the 
P^' c ^j et -t, there was a consider.' able amount of pressure from the 
to get the materials out, into the schools. There is 
a difficult decision as to when educational materials are 
good enough to prove useful and yet not bad enough to promote 
eternal rejection by educators of the whole enterprise. Does 
one hold off on distributing materials until they are really 
polished, or does one distribute them with disclaimers? One 
of the key assumptions of the Instructional Objectives Ex- 
change is that over time a truly praiseworthy collection of 
objectives could be assembled. We opted for the more auda- 
cious approach, distributing materials which we recognized 
were far from satisfactory. We asked the indulgence of the 
user at the same time we solicited from him the reactions 
which would permit us to improve the quality of our collec- 
tions. Time will tell whether this strategy has been aopro- 
priate. ' 
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, . — - - P rocedura ^- -Question . Because the whole basis of the 

Objectives Exchange approach hinges on the educator's selec- 
tion, rather than generation, of objectives, it is necessary 

t f > _ c ? nsider tbe appreciate form in which the objectives 

snouxd be presented for that selection operation. While the 
Exchange was focusing on the collection and development of 
objectives, we faced a key procedural question. In what man- 
ner should the objectives be stated? Just how precise must 
the objectives be? Should sample items be given? One acti- 
vity early m the project's existence dealt with this issue. 

Anyone who has worked much with operationally stated ofo-sc- 
tives realizes that one can state an objective behaviorallv 
with varying degrees of specificity. But increased speci- 
ficityfvmiCaxxy requires more words. And since we wish 
busy educators to read these objectives, perhaps too many 
w£f? S n Per ob 3 e ?tive would be aversive. Maybe a shorter, some- 
■ specific objective plus a sample test item reflect- 

ing tha.t objective wouid. be pref Birred. 



According 1 f°r the topic of first grade mathematics (a 
topic understandable to most teachers) six objectives were 
formulated^ each in two versions. While both were stated 
behavioraixy one was very specific , while the other version 
of the same objective was only specific . For example, the 
very specific version of one objective was as follows: 



Given two pictures, one of which is 
composed of items which are designated 
a set , and another which combines 
members of that set with other ob- 
jects, the student will be able to dis- 
criminate on the second picture those 
objects which belong to the set by 
circling them. 

The sp ecific version of that same objective was the following: 

The student will be able to select 
from a group of objects those objects 
whi-cdi have been included as members of 
a given set. 



Three of the six objectives in their very specific form were 
randomly assigned to one rating sheet along with the other 

f ° rT ? 1 ? b ^ ectives - A second rating sheet con- 
f the remall ? :L1 ?g objectives, that is, three specific 
y - XS * The order in which the objectives 



were placed in each rating form 



was 



also determined randomly 



A third and fourth rating form were developed by addina a 

Srina^oV ^ 6 " 1 fol;Lowing each objective. For instance, re- 

sS£le g -i^m he pr ? vxo * s b y° objective examples, the following 
sample item was placed below each: y 
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Thus, four ratine/ forms were available, two without sample 
items and two with sample items. At the top of each form 
the following identical instructions were given: 

Please rate each of the objectives on 
this page in terms of the manner in 
which it is stated. Do not concern your- 
self with the content of the objective, 
only attend to the form. Rate an objec- 
tive according to how easily and clearly 
you understand it, so that you might be 
able to decide whether you would wish to 
select the objective for use in your own 
instructional situation. Write the num- 
ber of the appropriate rating in front 
of each objective: Excellent-5, Good-4, 

Average-3, Fair- 2, Poor-1. 

It should be apparent that we were attempting to discover 
educator preferences for specific versus very specific objec- 
tives, both with and without sample test items. By having 
teachers rate objectives on two of the forms, for example, 
we could compare preferences for the specific or the very 
specific version. 

Shortly before the project commenced, the four forms were 
randomly distributed to 221 Minnesota educators (teachers, 
supervisors, and administrators) at the conclusion of a two- 
day conference of the Minnesota Association for Supervision 
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and Curriculum Development (ASCD). Approximately 50 parti- 
cipan ts filled out each of the forms . Their ratings supported 
tne very specific versions in 10 of the 12 possible contrasts 
(on one there was a tie), It had seemed possible the;, rn 
mating objectives with icains , the less specific objective s 
might be preferred. Such was not the case. But since th^ 
forms had been distributed at the close of a two-day session 
which the writer had been the principal speaker and 
had attempted to encourage the use of measurable object!’ as, 
these made methodological problems. The whole thrust of the 
two-day session had been to applaud specificity. Clearl* 
the data were tainted. Replication was in order. 

Fortunately, the writer was again addressing an ASCD grouo 
several months later in California and a similar data-gather- 
lng venture was undertaken, this time at the beginning of 
the session, prior to any advocacy of specificity. Approxi- 
mately 150 California ASCD members (teachers, supervisors 
and aoministrators / , meeting at a two-day conference were 
randomly given the four rating sheets. All of the ratings 
(5 ~ excellent, 1 - poor) are presented in Table 3 for both 
the Minnesota and California educators. 



Several analyses are of interest. First, it will be noted 
tftat although still preferring very specific objectives, the 
■ i, ornia educators only selected them in eight of the pos- 
sible 12 contrasts. 



Second, if one compares the California versus Minnesota rat- 
ings on each objective, the degree of disparity in favor of 
„ e y - er V J aPscif ic objective form emerges even more clearly. 
j?or example, without a sample item, objective number one in 
At f - er y specific form received a 3.6 mean rating in Minne- 
sota and a 2.7 mean rating in California, the difference of 
mJ favoring Minnesota. However, in the specific form, the 
same objective was rated 3.4 in Minnesota and 2.9 in Calif- 
ornia, the difference of .4 favoring Minnesota. The dispar- 
ity between .9 and .4 is .5, reflecting a stronger preference 
for the v ery specific objective. These differences between 
differences were statistically significant in favor of the 
SEScific versions (Wilcoxon matched-pairs signed ranks 
test) beyond the .05 level (two tailed). 

The preference for very specific objectives was sufficiently 
strong to warrant the use of the very specific rather than 
specific form. It was decided to employ that form during 
early development activities. 



DEVELOPMEN T DURING THE MIEOLE PHASE OF TH^ PROJECT 

As the project entered its intermediate phase and the pres- 
sure of getting out the initial collections subsided some— 
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*(vs) very specific objective *(s) specific objective 



what, an attempt was 
the development of r 
chapter details the .... 
to the development z 
The questions to he . 
objectives be secure; 
jectives be organi zee- 
first draft objecti- 
external reviews be ■ 
pared for the ob jeciz 



-e to be somewhat more systematic in 
- ollections. This section of the 
actions given to staff members assigned 
ew collections of objectives and items, 
zted are: (1) From what sources can 

- generated? (2) How should the ob- 
;; a collection? (3) How should the 
— -e generated? (4) How often should 
zred? (5) How are test item pools pre- 



Sources of Objective^ 

aR — vidual assigned to prepare a collec- 

ob 3 ec tiv e s w; probably be to decide where the ob- 

is S thiSr^fr 011 - ' Perhaps another way of putting it 
tiveS Si i f tlle p ~ £ 53 wbere °ne can either obtain objec- 
balorf or ' perha P s ' tb = content on which objectives can be 

lowina u " d ^ btedl y more such sources than the fol- 

lowing, but at least these should be considered. 

Obneo tr^o i ^ Ut w d C o \ lftC t in ^ - si nce 1968, the Instructional 

out the Jouf? 0113 ? 9 ® ha£ ^f en enc our aging educators through- 
out the country to contribute objectives to the Exchanae If 

there are any such collections in the Exchange which have al- 
£o 3 st ? ontribub ? d ' this would obviously be a good pl a Se 

Of i? n f° me flelds there will be a respectable number 

of such collections. For others there will be few, if any? 

whl>h Textbooks A second source of content from 
SoS eS Ca S i? fe *red ™>uld be the standard text- 

SfS ? aed in ^ he subjects and/or grade levels for 

?^ leCtl °? ir intended. Such textbooks are char- 
acteristically available at a university or colleqa curricu- 

i^ividSal^^ ° r - at cr - iar comparable resource centers. The 
to d notf th^^S 3 ^ 1119 t “ e objectives can consult such texts 

oSen?e can r J n ? s . ° f topics covered and, as a conse- 

quence, can infer what rypes of objectives might be yielded 

^.consequence of the student's interactional th such con- 



Curriculum Guide s. Many school districts have cretJarpH 

cSntafn^sSuc^on 0 ?"^ S ? llab± ' or other^t^lfSSS** 

ilv availablf ?? 1 oh ^ ct ^ ves - ®ese guides are ordinar- 

UCLA fc T a ^ y curr:Lculuin resource center such as the 

neaSoiast^t ° r in the e “"iculum «* 

Possi l l £ f oS? i0 ^ al Group -^commendations . Another source of 
possible objectives are reports of professional teachers' 
associations such as the National Council of Teachers of Eng- 
lish. oome times these gr- o s have assigned committees to the 



task of making recommendations concerning what ought to be 
^ "^ n schools . * Often the recommendations will be 

stated m a form too general for instructional purposes, but 
suitable for guiding one attempting to prepare new collec- 
tions of objectives . Consult also the publications of the 
subject matter professional associations, not associated with 

education, for example, the American Psychological Associa- 
exon. 



Cu rricuJ-um Projects . Since the 1950 's we have seen a 
number of new curriculum" projects funded in this country 
by various philanthropic groups and the federal government. 

If there is a project, for example, BSCS, SMSG. etc., which 
bears upon the collections being developed, then it would ob- 
viously be advantageous to secure whatever reports exist from 
such a group in order to see if the project has offered gen- 
eral or specific advice regarding the goals which should be 
pursued for that field. 

a rSal ganger that by relying on extant collections 
f objeccives and somewhat out-of-date textbooks that a sta- 

° f ° Ught t0 be tdU 9 ht vd.il be produced, 

ihis might be countered by consulting such grouos who have 
been commissioned to make recommendations regarding current 
objectives, for their vision regarding the future will char- 
ac t er i s t ic ctll_y be somewhat more enlightened. 

Prj acticinq Teachers . Another group of individuals who 
can say much regarding what ought to be taught in particular 
f i e ‘ Ld f • are ths te achers who are engaged in day-to-day in- 
struction. If the developer of a collection has any teacher 
acquaintances in the subject or at the grade level of inter- 

? e highby useful to get their views regarding 
what ought to be taught. This can be done either informally 
or, perhaps, more systematically by employing paid consultants 

P r f? ticin g teachers, one must remember that 

What they are cur rently teaching is 
what sh ould be taught. The collection developer may have to 
encourage them to set aside their perceptions of practical 

in' 1 - moiS ?i an< ^v^° ^ ie ™. the Pursuit for defensible objectives 
xn r_\ more flexxble fashxon. J 

, pedagogical Journals . There are some quasi-popular jour- 
nals pubiished for teachers, e.g. , Grade TeLher , l£ s t?uctor 
w ic may have articles included that suggest possible objec- 
tives. By consulting the Education Index , a guide to peri- 



j • * For example - the Association of Teachers of Social 
Studies of the City of New York. Handbook for Social Studies 

y 5a T^ 1 9 67 Edi tion t Holt, RinehartT” w!niton7 _ Inc77Ni5 § ' 
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odical literature in education, the developer may be able to 
isolate some titles of articles which appear promising. 

Academicians . While practicing teachers are conversant 
with the subject matter at hand, they are often too busy to 
keep up with recent developments in their field. As a conse- 
quence. it may be advantageous to consult college or univer- 
sity academicians who are specialists in the field under con- 
sideration. They may be able to offer advice regarding what 
should be taught in that field. 

For Affective Collections . Most of the sources identi- 
fied above are primarily for cognitive objectives collections. 
If the developer is commissioned to devise an affective or 
psychomotor collection, there will obviously be different 
sources. For instance, certain affective dimensions may re- 
quire one to search the sociological or psychological liter- 
ature. Often the standardized tests described in one of the 
Buros 1 Mental Measurements Yearbooks may give some ideas re- 
garding how to design measures. For indirect approaches, the 
text* by Webb and others which describes unobtrusive measur- 
ing strategies should certainly be consulted. 

These are not meant to be exhaustive sources of content for 
a new collection. Generally, the person developing the col- 
lection will have already had some conversance with the field 
and will have his own ways of isolating possible objectives 
that should be covered in that field. Additional sources 
are, therefore, thoroughly appropriate. Remember that the 
purpose of developing collections of objectives is to make it 
easier for an educator to organize his instruction around 
measurable objectives. We do not wish to dictate what objec- 
tives should be selected and, thus, should always present a 
wider range of objectives than would be typically selected 
by any one educator. This means that the particular bias of 
the person preparing the objective collection should not con- 
strain the objectives too narrowly. We wish to include ob- 
jectives which might be chosen by educators who are of a dif- 
ferent persuasion than the person or persons involved in pre- 
paring a given objective collection. On the other hand, we 
would not include thoroughly aberrant objectives merely for 
their atypicality. But to get a general fix on a field, the 
use of sources such as those outlined above may be useful. 



Organization of the Objectives 

In general, materials being developed under this project are 
currently using a grade and subject level classification sys 



*Webb , 
Research in 
1966. 



E . J . , et . al . Unobtrusive Measures i Nonre active 
tlie Social Sciences . Rand McNally and Co. , Chicago, 



tem. As a consequence, most of the objective collections 
fall within a given subject matter, for example, history, and 
also at particular grade ranges, for example, 4-6. The de- 
cision to use a range of grades rather than a single crrade 
was dictated primarily by the fact that in different parts 
of the country certain objectives which would be suitable at 
one grade would be inappropriate for the same grade else- 
where. By organizing the collections in the following grade 
ranges: K— 3, 4-6, 7—9, 10—12 (although some collections 

cover wider grade ranges) , we offer the user more latitude 
i^* his selection of suitable goals. If the grade range de- 
cision has not been made prior to the preparation of the ob- 
collection, you should offer a well— considered recom — 
mendation regarding this question as soon as possible. 

Early collections produced by the Instructional Objectives 
Exchange did not even contain a table of contents and, as a 
consequence, educators had a great deal of difficulty in sim- 
ply using the booklets. At the very least, we need some kind 
of organization within the document so that the potential 
user can more readily employ it. For the most part, the ob- 
jectives have been organized around topics, although alter- 
native organization schemes are possible, e.g. , themes, prob- 
lems, etc. Perhaps the most important point is to organize 
the objectives in such a way that the typical user of those 

objectives in the field will find the collection easy to util- 
ize . 



First Draft Objectives 

After the initial exploration and decisions regarding how 
the collections should be organized, there is the necessity 
to put together a first draft of an objectives collection. 

It is suggested that prior to engaging in an elaborate amount 
of work related to this task, the person preparing the col- 
lection should describe the organization scheme to be used 
and then give a limited number of examples of objectives for 
that organization. Perhaps several objectives for each major 
classification rubric can be prepared. For instance, if 
^ lere are eight major categories in the organization scheme, 
then maybe two objectives per category can be prepared. This 
scheme can then be presented to someone else for review, so 
that an inordinate amount of energy is not expended on an or- 
ganization scheme which is clearly inappropriate. As will be 
suggested later, the necessity for securing many other people's 
reactions to the emerging work is particularly important in 
producing new objective collections , for in these largely un- 
mapped lands one can move in an incorrect, direction and thus 
waste many hours producing objectives that are indefensible. 

. Genera.lj.ty Level . One of the more difficult questions 
facing the field of instructional objectives as a whole is 
how to resolve the level of generality issue. On the one 



O 

ERIC 



26 



33 



hand, we want objectives that are clear and that accurately 
communicate an educational intent. On the other hand, we do 
not wish to have a thousand objectives in each collection 
thereby insuring that no one will use tnem. Therefore we 
must certainly be attentive to the kinds of content gener- 
ality advice which Professor Baker* and others have offered 
The objectives ought to be such that they cover a wide ranqe 
of earner behaviors rather than a single test item. The 
last thing we want to produce is a set of text item equiva- 
lent objectives (i.e., wherein an objective, though measur- 
ablo, is essentially the same as a single test item), since 
such objectives would obviously be too numerous to be manage- 



At the moment, however, the resolution of the generality 
level issue is not in sight. Perhaps the best guidance that 
can be supplied at this moment is to avoid test item equiva- 
ient objectives, but make certain that the objectives are 
still measure able and will communicate with relatively little 
ambiguity to users regarding what learner behaviors are sought. 
In other words, avoid sins of excess in either direction. 



. . Termin al Objectives Sought . Once more to avoid objec- 
lves too numerous to be useful, we are concerned with gen- 
erating terminal rather than en route objectives in the Ex- 
change. While it is true that teachers may have to accomp- 
lish a number of preliminary objectives in order to get 
learners to achieve those presented in our objectives collec- 
tions, these en route objectives should not be included in 
the collections. There would simply be too many objectives. 

„ gjg^Leve ; 1 Objectives. In general, we are attempting 
to produce exemplary collections of objectives which educa- 
tors can use as models for guiding their instruction. There- 
fore, we certainly do not wish to produce objectives which 

d^m^ e ?^ m;L S^ n i ly the mOSt trivial level of the taxonomic 
J which a given collection is centered. We must not 

ma 2 y l0W level knowledge objectives, but instead 
^hould strive for as many high cognitive level objectives as 



Although affective objectives are eminently appropriate for 
any collection, one should avoid the iiaclusion of affective 
objectives which would relate to a broad range of subject 

4 - for example, assessing the learner's atti- 

tude toward a subject matter. Such objectives can be pro- 

oUSS i in a mo ^ e . gel ? era l collection devoted exclusively to 
such learner behaviors. 



, . . e P erson unfamiliar with technical questions associa- 

^?+*>, W i£ h p f e P^ ra ^ lon of objectives should acquaint himself 

st fJ nd f rd references on objectives, e.g. , Popham, w.J. 
Hall Ba i970 E * L ' ' ^ tablishinc T Instructional Goals , Prentice- 
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Securing Reviews 



It is important to get other people to react to the materials 
as they are being developed. This includes staff members 
within the project, teachers, professors, indeed, anyone who 
can be located who might have something useful to say about 
the collections. Generally, each person preparing objective 
collections will have a supervisor and this individual should 
be contacted at least every two weeks or so to monitor the 
direction of the development activity. The project does have 
funds to expend on more systematic reviews and, when appro- 
priate, we should definitely be paying high quality people to 
look over a collection which is in the process of being 
formed. The more systematically these critiques can be pre- 
sented, the better. The developer should try to structure 
the response of the external reviewer so that the reactions 
offered are of maximum utility. 

As indicated earlier, it would be useful to have someone 
scrutinies the initial organization (including a few sample 
objectives) of an early collection. Later on, it would be 
extremely important to have some external reviewer check 
the complete range of objectives which have been prepared. 

Prior to an objectives collection's final development so that 
it is available for distribution, we. must have a final check 
from some knowledgeable individual (such as an intelligent 
teacher working in the grade range specified) so that we do 
find ourselves making available indefensible material. 



Preparing Items 

After the objectives have been approved, it is then appropriate 
to invest the energy necessary to produce the requisite num- 
ber of items for a collection. At the moment, we have been 
preparing one sample item and five additional items, that is, 
a total of six for each objective. It seems imprudent to 
produce such items prior to the approval of the objectives 
since certain objectives may be eliminated. On the other 
hand, it might be necessary to produce one test item to com- 
municate more fully to a reviewer what the objective is sup- 
posed to accomplish. Usually this would not be the case, 
since the objective itself should be sufficiently clear to 
satisfy the communication requirement. This very point, in a 
sense, leads to the perplexing problem of how to prepare homo- 
genous test items which are congruent with the objective they 
are supposed to measure. This discussion can be initiated 
W-^-'fch the candid admission that here is an area where the tech- 
nology is currently quite primitive and where we will be doing 
a f a i r amount of stumbling in the next several years. 

Nevertheless, we are using the approach of Wells Hively and 
his associated at the University of Minnesota who have given 
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as much thought as anyone to considering schemes whereby homo- 
genous sets of test items may be generated to assess the at- 
tainment of a given objective. It should be pointed out, 
of course, that "test item" is being used very broadly to in- 
clude any testing procedures, not just a paper and pencil 
test. The scheme that Hively and his colleagues have pro- 
duced is called domain - referenced achievement testing . 

If an objective is stated in measureable terms yet is also 
somewhat general in form, then diverse types of test items 
can be produced from it. We can think of such an objective 
as a measureable but, in a sense, inexplicit objective. On 
the other hand, one can think of an objective which carries 
with it so many limiting phrases that there is no question 
at all about the kinds of measures which would be designated 
to assess its attainment. Yet, such elaborately stated ob- 
jectives would probably be of little utility to teachers 
since they would take too long to read and would, undoubtedly, 
frighten off the potential user. 

As a consequence, we are using the item form scheme which 
Hively proposes. An item form is something like an inter- 
mediate set of delimitations which accompanies an objective 
and is supposedly of use to the person preparing the test 
items for that objective. The item forms which we are cur- 
rently employing must be brief enough so they will be use- 
able for our own staff, yet should contain the ingredients 
necessary to reduce the variability of items that would be 
produced to assess the object ive. There are three elements 
in the item forms we have recently begun to employ: (1) in- 

structions to students, (2 ) stimulus limits, and (3) response 
limits. A little elaboration regarding each of these elements 
may be of some value. 

Instruc ti ons to Students . Generally, test items used 
to assess an objective will be One of any number of standard 
forms, for example, multiple-choice, fill-in, etc. It is 
important to know which kind (or kinds) of items are to be 
employed. This certainly permits the possibility of more 
than one type of item. If there are any special kinds of for- 
mat for the items, this should be indicated in the instruc- 
tions to the students. In other words, if there are things 
the students ought to know other than what will be supplied 
in the actual item, this should be explicated in this part 
of the item form. Generally, this will be a very brief sec- 
tion of the item form. 

Stimulus Limits . This is the really tricky part of the 
item form. Ideally, this section would delineate all the 
possible stimuli the students would encounter in any item. 

For example, in a mathematical problem the stimulus limits 
section would indicate the possible numerical combinations 
the student would receive in the stem of the item, and in a 
selected response item in the alternatives from which he is 
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to select. Indeed, most of Hively's early work concerns 
item forms involving mathematics and the sciences. On the 
other hand, one can think of a history ob? active in which one 
might want to describe all possible content such as inter- 
national conflicts which could be used as eligible elements 
in test items designed to measure the objective. Yet, to 
identify all eligible contenders for test items would un- 
doubtedly be too time-consuming, so it wou 1 seem to make 
more sense for us to isolate several classes of content or 
types of problems (or whatever the objective deals with) 
which could be used in test items. By trying to identify 
some examples of eligible elements we may be able to move 
toward clarity to the desired degree. 

On the other hand, if it is possible in one or two phrases 
to identify the essential attributes which would constitute 
any eligible stimulus elements , then this should surely be 
done . 



Response Limits . This section should indicate whether 
a student is to make his response by selecting among alterna- 
tive choices (which would be defined in the stimulus limits) 
or would be constructing his response and, if so, whether 
there would be any constraints such as the room available 
for the response, the time available for the response, the 
amount of assistance available (reference books) for the res- 
ponse, etc. 

In addition, it is necessary for the person constructing the 
item form to indicate precisely how one is to judge whether 
the responses of the learner are correct. Particularly in 
the case of a constructed response where there is more poten- 
tial variability in the way people respond, it is crucial to 
include such criteria. For example, if the student is pre— 
P a ring an essay describing a particular phenomenon, what 
critical elements must be included in the learner's descrip- 
tion in order for it to be considered satisfactory? 

Even with selected response items it is important to identify 
the criteria for judging acceptable selections, since very 
frequently this will sharpen the item writer's perception of 
how the item should be constructed. It is insufficient to 
simply say that a response is correct by stating it is "the 
correct answer. " 



A Need for More Systematic Guidelines 

As our staff continued to develop objectives collections dur- 
ing the middle months of the project's existence, it became 
clear that a more rigorous set of guidelines for the develop- 
ment operation was required. Accordingly, we devoted consid- 
erable thought to the formal structure which should guide 
such enterprises. The position taken at the close of the 
project is presented in the next section of the chapter. 
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DEVELOPMENT DURING THE FINAL PHASE OF THE PROJECT 



This section of the chapter is based primarily on a scheme 
evolved by E. B. Buck and, while it treats a model primarily 
associated with development ( subpurpose number one of the 
project), it relates to the quality control activities as- 
sociated with subpurpose number two of the project. 

^ I ^ ie . •*- < pll°wing paragraphs present a detailed outline of the 
activities and decisions involved in the development of col- 
lections of instructional objectives. Specifically, a de- 
cision model of the development process will be presented. 

. means that each significant step (decision or activity) 
m txie development process will i>e descrit>ed according to: 

1; the nature of the activity or decision 

2) the person or persons responsible for the activity 

or decision 1 

3) the resources likely to be used 

4) its relationship (sequence) to other activities or 
decisions 



Furthermore , for each decision, the criteria governing that 
decision are specified. Hence, the decision model can be 
thought of as a detailed map, including rules, of the devel- 
opment process associated with the preparation of objectives 



A diagram of the decision model is shown in Figure 1. The 
arrows (- f ) indicate the sequence; rectangles (dl ) indicate 
activities; and diamonds (O ) indicate decision points. The 
person responsible for engaging in the activity or making the 
decision is shown in parentheses. A detailed descriptioS of 
activity and each decision follows the diagram. Since 
collations will periodically undergo revision, a discussion 
of how revision fits into the decision model is included in 
a subsequent chapter. 



Dl: 



Should thi ± collection be developed? 



The initial decision (Dl), which in essence instigates the 

5 r ° CeS ^' ±S w £ etl ? er or not a given subject matter 
sh< ould be used as a basis for a collection of objectives. 

asked^ 11 ^ thlS decls ion, the following questions must be 



Is the subject area included in any other available 
collections? 

Is the collection likely to be used? (Is there a 
demand for this proposed collection?) 

Are there project personnel available for working 
m this area? 
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PROJECT DECISION MODEL 
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Will the lection be fairly easy, quick, and 

straigt .-forward to generate? (Are there con- 
tribute!; collections in the project files 
which v.; — 1 give us a head start?) 

Can the serjsct conceptually be taught at the 
level _r_volved? 

Obviously, when several areas are proposed, and when resources 
(manpower, funds and. time) are limited, a cost-benefit analy- 
sis of the alternatives is needed- 

The responsibility for this decision rests with the direc- 
tors and management of the project. Developers and staff 
members are, of course, encouraged to investigate and suggest 
new areas for collections. They should, however, be able to 
justify any proposed subject matter area in terms of the 
above questions. 



Possible sources of data and information for reaching this 
decision are: 

Structure of knowledge literature 
Analysis of content of current collections 
Availability and specialty of project personnel 
Analysis of school curriculums 
Consultation with school personnel 



Developing the structure or organization of the proposed 

collection . 

Once the decision to generate a collection in a given area 
has been made, the next activity (A.1) is to formulate a struc- 
ture for the subject matter or desired student behaviors. 

In essence, this means identifying major topics and behaviors, 
then determining their relative importance for the given age 
level. It is necessary to give weight to areas since these 
will act as a guide in determining the number of objectives 
to be included under a given area. If weights are not used, 
then the number of objectives included under each area is apt 
to be in proportion to the ease of writing objectives and/or 
items in that area. In the extreme, we could end up with a 
ma 3 or "ify of the objectives in a collection in an unimpo: tant 
area. Not only does this place a hardship on the user (he has 
to select among them! ) , but it also tends to discredit the 
validity and representativeness of the entire collection. 

The person assigned to develop the collection is responsible 
for the identification of the structure and the determina- 
tion of weights . 

following sources of data and/or information might be 
useful to the developer in performing this task: 
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Materials in curriculum lab 
Published textbooks in area 
Current grade textbooks 

Consultants (teachers, experts in field) 
Professional group recommendations 
Journals in field 
Curriculum projects in area 



D2: Is the structure adequate to meet project needs? 

Once a structuring of the subject matter and learner behav- 
iors has been made, it is necessary to determine if this pro- 
posed structure is feasible. The following criteria, as a 
minimum, shotild be used: 

The structure should be readable and logical. 

The structure should be such as to facilitate the 
location of objectives by users. 

The structure should be such that it will complement 
a variety of curriculum schemes. 

All major topics of the field should be covered, or 
justification for the exclusion of topics should 
be presented. 

The categories of behaviors should be higher than 
the lowest level (Bloom's Taxonomy ) . 

A justification of the weights should be presented. 

If the proposed structure fails to meet the criteria, then 
it will be necessary to restructure the collection (i.e., re- 
turn to activity 1). 

It is the responsibility of the developer, who proposed the 
structure, to justify that structure according to the above cri- 
teria. The final decision to proceed or revise, however, will 
be made by project management. 

In justifying a structure, or determining whether one is ade- 
quate, it will be necessary to obtain the opinions of sub- 
ject matter experts. Also useful would be the following: 

Any of those sources listed under activity 1 
Educational experts (project staff, teachers, stu- 
dents, administrators, etc.) 

A2: Writing of objectives and sample items 

Activity 2, the actual writing of objectives and at least 
one sample item per objective, is perhaps the most time con- 
suming of all the steps on the development process. The pur- 
pose of this activity is to write objectives which reflect 
the scope and depth of the agreed upon structure. The number 
of objectives for each topic should be in proportion to the 
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relative weight assigned to that topic. Each objective 
should be accompanied by a minimum of one item. hater on, 
more items for each objective will be written. Since the* 
time necessary to write items is often considerable, it is 
advantageous to have objectives with one or two items re- 
viewed by experts before the developer invests a lot of time 
in item writing. In the cases of affective measures, where 
o jectives are accompanied by a measuring instrument, it will 
be necessary to include enough items to give a representative 
sample of the measure. 



The developer of the collection and any staff personnel as- 
signed to the project are responsible for the development of 
both objectives and items. The division of tasks among pro- 
ject members is left to the discretion of the developer. 
Sources of possible objectives, or ideas for objectives * are : 



contributed collections 
standard textbooks 
curriculum guides 

professional group recommendations 
curriculum projects 
pedagogical journals 



D3 : 



objectives well— stated and clear? 



At this point, when 
completed, we must, 
rect. As a minimum 
s idered: 



a first draft of the collection has been 
ask if the objectives are technically cor- 
, the following criteria should be con- 



Objectives should be behaviorally stated. 

Objectives should be terminal as opposed to en route. 

Objectives should predominantly be at the higher 
levels of the taxonomies. 

The language in the objectives should be nrecisp 
with vocabulary well defined. precise. 

The objectives should be appropriate to the grade 
level. 

The language in the items should be approp late to 
the grade level. 

The items should measure the objective without 
being trivial. 

The number of objectives under each topic or behav- 
ior class should be proportionate to the weight 
determined in activity 1 (determining the struc- 
ture of collections). 



If the objectives do not meet the criteria, it will be neces- 
sary to return to activity 2 (writing objectives). 

^Sr, r ? V:LeW ob j ec tives is to be conducted by educa- 

tional consultants or staff members not involved in the actual 
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writing of this collection. The decision to proceed or re 
cycle is their responsibility. 



Do the objectiv es adequately refle ct the subiect matter 

and weightings? 

The last decision point consisted of looking at the objec- 
tives fiom an educational standpoint. Now, they must be con- 
sidered from a subject matter view. In essence, the collec- 
tion must satisfy the following criteria: 

The entire domain of subject matter as specified 
in Activity 1 should be covered. 

All objectives and items should be technically 
correct and unambiguous . 

The behaviors called for should be behaviors 
exhibited by someone working in the field. 

The number of objectives under each topic should 
be in proportion to that topic's importance. 

If the collection does not satisfy the criteria, it will be 
necessary to determine whether the problem lies with the struc 
ture of the subject matter (activity 1) or with the writing 
of objectives (activity 2), and then return to that activity. 

An acknowledged subject matter expert should review the col- 
lection and decide if the collection adequately covers the 
field. 



A3: Item Writing 

After approval of objectives and items by the subject matter 
expert, the developer should proceed to generate additional 
items for each objective. An objective should have a minimum 
of six items. The preparation of items is under the direction 
of the developer . 



D5: Review of Items 

Just as the objectives with one item underwe.it extensive re- 
view by experts, so should all the items generated in activity 
3. Specifically: 

The language in the items should be appropriate 
to the grade level. 

The items should measure the objective without 
being trivial. 

items should be technically valid and unambig- 
uous . 
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Both educational experts (project director (s) and 
subject matter experts will be involved in this 



staff) 

review. 



and 



Should field testing be conducted? 

colWtiX^ ° f Hj® collection by experts, most cognitive 
are then prepared for immediate publication 
Wxth the affective collections and occasionally wiS Si cog- 
txve collections, it will be necessary to field test items 

testing would be necessary to ob- 
rollowing information: teacher, student, or 

tiS^ tings of objectives; teacher evaluations of objec- 

Svllua?^n c ^° ns ^ r ? m sub ^ ct ma tter Standpoint; teacher 

of usefulness of items; and information on student 
performance on items or inventories. If field testing is 

If C n2t ar £ho th ?£ test designing (A4) will be the next Step. 
(A6? ' h S collect;LO?:i can Prepared for publication 



The decision to field test or not will be made 
management; however , the developer should make 
tion and provide t vidence to support it. 



by project 
a recommenda- 



A4: Test Design 

When it is necessary to do some form of field testincr *-h^n 

t£at n ?^H iV i ty in proceLis?op?anfS? 

testing* The plan should include the following: 

A statement of the purpose and specific questions 
to be answered through testing. 

A description of populations to be tested (students, 
teachers, etc.) including general characteristics 
of the district. 

A statement of what activities will be expected of 
all participants. 

A description of how participants are to be selected 
( i • e. , sampling plan), 
time table for major steps. 

statement of how the information (results) will 

be used to revise or make decisions about the col- 
lection . 

Prototypes °f materials to be used in testing. 

A description of the type of analysis to be per- 
formed on the data. 



A 

A 



The test design can be worked out by the 
collection or by a statistician or both, 
available on the project staff to aid in 
sis of field tests. 



developer of the 
Statisticians are 
the design and analy— 
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D7 : Will the plan answer the specified c~Jt r l- ms? Are all 

questions that should be asked being asked? 

Once a plan of testing is proposed, it is necessary to care- 
fully examine it to assure that the plan satisfies the pur- 
poses of testing. This will assure that the time, effort, 
and expenses of testing prove profitable. specifically, the 
following questions must be asked about the plan: 

Do the specific questions being asked provide 
enough information for decision making? Are 
there any other questions (purposes) that 
should be considered? 

Is the sample size appropriate? 

Is the sample representative of potential users? 

Is the load per person unreasonable (i.e. , are we 
asking the subject to spend too much time?) 

Are data collecting procedures well defined? 

Are the proposed analysis techniques appropriate? 

Is all the information necessary being collected? 

Is the prototype material appropriate? 

Testing cannot proceed until the plan answers all these ques- 
tions. Staff statisticians and management will determine 
the adequacy of proposed plans. 



A5 : Carrying out of Test F an 

Once the test plan is approved, implementation can be started 
immediately. Activity 5 includes: preparation for testing, 

collecting data, analyzing data, drawing conclusions and 
■writing the final report. The final report should include: 

Summary of testing plan and procedures. 

Main results of the analysis. 

Implications of the analysis fox' purposes of 
revision or other decision making. 

Suggested course of action (i.e. , revise, re- 
test, redevelop, drop project, etc.) 

These activities are carried out by the developer with staff 
support as needed. 



D8 : Is revision or retesting of the collection necessary? 

The field test results should now be used to decide the next 
course of action. The alternatives are: publish, retest, 

revise objectives or items, revise structure (hence also ob- 
jectives and items ) , or drop the project. In deciding among 
the alternative actions , the following questions must be 
asked: 
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Do the data support the conclusions (recommended 
action ) ? 

Is the importance oi the revision (if revision is 
necessary) such as to justify the additional 
cost? 

What is the probability that added -work on the col- 
lection will significantly improve the collection? 

Project management is responsible fox: making this decision 
and selecting the alternative to follow. 



A6: Prepare Collection for Distribution 

Now the collection is "technically" ready — that is, it has 
met subject matter and project specifications. The final act- 
ivity before publication is to add all the introductory in- 
formation and tables to make the collection usable. Specifi- 
cally, the following should be included: 

An introduction: description of project, its pur- 

poses and aims , discussion of how to use the collec- 
tions; rationale for the collection; results of 
testing if pertinent. 

A table of contents. 

A credit page: give credit, to developers, consult- 

ants, and staff members who worked on collection. 
References to qualificavi on and associations 
desirable. (Example: Ur.. Joe Dlow, Ph.D. , 

Professor of Garble, U.C.2..A. ) 

Bibliographies, Glossaries, or cross-reference tables 
if subject matter calls for it. 

At this time, it is also necessary to determine the format 
f Q3 - objectives and items. Although to date a fairly standard 
format has been used, this can be varied if such would improve 
the usefulness of the collections. When the developer feels 
the collection is ready for publication, he should present it 
in final form for review to management. 

final preparation of the collection is done by the dev- 
eloper with staff support. 



: Is the collection ready for distribution? 

This represents the final review before a collection is 
placed in the public domain. Specif xcally , it will be re- 
viewed to determine if: 

An introduction is included and adequate. 

The table of contents is included and easily read- 
able. 

A credit page is included. 

All revisions have been made. 
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The format is readable and clear. 

Each objective has at least six items. 

The final clearance of a collection for distribution will 
come from the director(s). 




- 42 - 



49 



CHAPTER 4 



T WENTY— EIGHT OBJECTIVES COLLECTION ; 
.DESCRIPTION AND PI S S EMI NATI ON 



product of this project was the development of 
collections of objectives and related measuring devices. B 
markedly expanding the number of such collections available 
to educators, the project has increased the likelihood that 
those who wish to employ an objectives-based approach to 
instruction and evaluation may do so without expendinq un- 
necessary energy in the creation of certain materials basic 
to such enterprises. 



In all, twenty-eight new collections of objectives and 
measures were developed as a consequence of this project. 

twenty-eight collections contained approximately 
2,000 instructional objectives and 7,000 test items. Because 
their development was the focal activity throughout the pro- 
ject s existence, a brief description of each collection will 
be presented m the remainder of this chapter along with the 
content topics included in each collection. 

Following the descriptions of the twenty-eight included col- 
lections , information will be presented regarding the dissem- 
ination of these materials during the past two years. 



THE COLLECTIONS 



Language Arts , 4-6 



D escription : Contents of this collection include structure 

and types of sentences, parts of speech, capitalization, 
punctuation, linguistics (word analysis), composition, 
literature. (194 objectives, 1 item per objective) 



Contents : 



Grade 4 



Gr aimfiar— Syntax 
Etymology 

Grades 4—6 



Grammar - Parts of Speech 
Grammar - Syntax 
Grammar - Phonology 
Grammar - Morphology 
Mechanics and Conventions 
Composition 
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Oral Language 
Reference Skills 
Literature 

Grade 5 



Grammar — Syntax 

Phonology 

Morphology 

Grade 6 



Grammar - syntax 
Grammar - Phonology 
Composition 



English Grammar. 7-12 

Description: This collection contains objectives and 

evaluation items for English grammar in secondary 
schools. Although there are some objectives which deal 
with the traditional approach, the main concentration 
is on the Roberts transformation approach. (84 objec- 
tives, 6 items per objective) 

Contents : Parts of Speech 

Syntax 

Morphology 

Phonology 

English Skills, 7—9 

Description : This collection contains objectives and 

evaluation items for the teaching of English skills in 

junior high school. (76 objectives, 1 item per objec- 
tive) 

Contents : Composition 

Mechanics and Conventions 

Diction and Tone 

Speech 

Mass Media 

Reference Skills 

Study Skills 



Enc lish Skills, 10-12 

P gsg.P i igt ion : This collection contains objectives and 

evaluation items for the teaching of English skills in 
high school. (37 objectives, 6 items per objective) 
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Contents: 



Speech 

Composition 

Mass Media 

Reference Slcills 

Mechanics and Conventions 



English. Literature, 7—9 

Description : This collection is designed to develop 

the students ' ability to analyze literature and to eval- 
uate its effects. (16 objectives, 6 items per objective) 

Contents : The Novel 



Characterization 

Motivation 

Conflict and Climax in Plot 
Theme 

Point of View 

Tone 

Setting 

Relation of the Novel to Experience 
Genre 

Poetry 

Characterization 

Genre 

Figurative Language 
Diction 

Repetitive Sound 
Tone 

Evaluative Process 



English Literature. 10—12 

Description : This collection is designed to develop 

the students' ability to analyze literature and to 
evaluate its effects. (34 objectives 4 . 6 items per ob- 
jective) 

Contents : Poetry 

Meaning 

Tone 

Diction 

Figurative Language 
Characterization 
Repetitive Sound 
Evaluative Process 
Genre 



Novel 



Point of View 
Genre 
Setting 
Tone 

Char ac ter i zati on 

Conflict and Climax in Plot 

Motivation 

Theme 

Relation to Experience 
Short Story 
Tone 

Conflict and Climax in Plot 

Point of View 

Setting 

Motivation 

Characterization 

Greek and Roman Myth 

Drama 



Conflict and Resolution 
Symbolism and Figurative Language 
Levels of Usage 
Theme 

Dramatic Construction 
Setting 

Characterization 

Non-Fiction 



Purpose 

Tone 

Evaluation 



Mathematics , 4—6 

Description ; This collection covers intermediate con 
cepts and skills. (233 objectives, 4 items per objec 
tive ) 

Contents r Sets 

Numbers , Numerals , Numeration Systems 
Operations and Their Properties 
Measurement 
Geometry 

Relations, Functions, and Graphs 
Probability and Statistics 
Application — Problem Solving 
Mathematical Sentences - Order, Logic 
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General Mathematics , 10— 1 2 



Description ; This collection emphasizes general con- 
cepts and skills. (123 objectives, 6 items oer objec- 
tive ) 

Contents; Sets 

Numbers , Numerals , and Numeration Systems 

Operations and Their Properties 

Measurement 

Per Cents 

Geometry 

Probability and Statistics 
Logic 

Applications , Problem Solving 



Business Educ a tion (Bookkeeping) , 10-12 

Descript ion ; This collection treats the basic procedure 
and concepts fundamental to the bookkeeping cycle. (17 
objectives, 6 items per objective) 

Contents ; Ten Bookkeeping Processes 

Bookkeeping Equation 

Balance sheet 

T— Account 

Journal! zing 

Posting to Ledger 

Trial Balance and Worksheet 

Financial Statements 

Closing Entries 

Post-Closing Trial Balance 

Tr suns action Process— Er, L ana t ion 

Banking Transactions 
Cash Transactions 

Cash Proof 
Petty Cash 

Payroll Records 
Depreciation 

Accrual and Deferred Expense 
Vocabulary Skills 



Business Education (Business Law), 10-12 

Description ; This collection is an introduction to the 
basic concepts and skills of Business Law. (37 objec- 
tives, 6 items per objective) 



Contents ; 



Foundations of Law 



Torts and Crimes 
Law of Contracts 

Legal Requirements 

Franchises, Copyrights, and Trademarks 

Assignments 

Termination 

Treatment of Minors 

Liability of Minors 

Elements of Fraud 

Offers 

Illegal Subject Matter 
Validity 
Sales Contract 
Insurance — Automobile 
Insurance 

Law o f Property 

Bailments 

Distinction Between Real and Personal 
Acqui siti on 

Limitations on Ownership 

Warranties 

Transfer of Title 

Mortgages 

Title 

Law of Negotiable Instruments 

Endorsements 

Essentials 

Defenses 

Law of Business Relations and Business 
Organizations 



Government and Business 
Agency — Corporation 
Agency 

Agenc ’ — Partnerships 
Voc abul ary 



Business Education (General Business) 10-12 

Description : This collection is designed to familiarize 

the student with some of the concepts which he will find 
most useful in the business world. (35 objectives, 6 
.items per objective) 
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Contents : 



Vocabulary 

Money and Banking Services 
Checking Accounts 
Cost of Living and Taxes 
Kinds of Business Ownership 
Sales 

Marketing Functions 
Insurance and Pensions 
Consumer Services 
Communications 

Employment Application and Interview 
Business Math 



Business Education (Secretarial Skills). 10-12 

Des cri ption z i'his collection emphasizes basic steno* - 
graphic skills*. (46 objectives# 6 i^eins per objective) 

Contents s Typing 

Dictation 

Composition 

Corrections 

Care of Typing Station 
Multiple Copies 

■v Common Business Forms 

Accuracy Percentage Table 
Rules for Counting & Marking Typo- 
graphical Errors 

Shorthand 

Brief Forms 

Dictation and Transcription 

Office Machines 

Telephone Systems 
Automatic Typewriters 
Tape Recorders 
Postage Meter 
Duplicating Machines 
Copying Machines 
Data Processing Systems 
Ten-Key Adding Machine 
Printing C 1 * \ator 

Full Keybc Adding Listing Machine 
Printing Calculator 
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Office Practice 



Behavior and Appearance 
Mail Procedures 
Personal Interaction 
Information Resources 
Travel Preparation 
Telephone Shills 
Preparation of Documents 
Filing Systems 
Filing Procedures 

Business English and Communication 

Business Letters 

Punctuation 

Grammar 

Word Usage 

Vocabulary 

Cover Letter and Resume 
Business Interview 
Pronunciation - Enunciation 
Public Speaking 



Home Economics , 7-9 



(74 objectives, 6 items per objective) 

Child Development 
Clothing and Textiles 
Consumer Practices 
Foods and Nutrition 

Home Management and Family Economics 



Home Economics, 10-12 

Description : (48 objectives, 6 items per objective) 

Contents : Child Development 

Clothing and Textiles 
Consumer Practices 
Design Principles 
He a J th S e rvi c e s 

Home Management and Family Economics 

Housing 

Pregnancy 



Auto Mechanics, 10-12 



Description ; 
Contents : 



Description : 



Contents : 



(185 objectives, 1 item per objective) 
Automotive Tune-up and Repair 



Electronics, 7-12 



Description s This collection is an introduction to 
basic concepts and skills of electronics. (50 objec- 
tives, 1 item per objective) 

Contents : Fundamentals 

Block Diagrams 
Direct— Current Circuits 
Primary and Secondary Batteries 
Magnetism 

Electromagnetic Induction 
Direct-Current Generators 
Direct-Current Motors and Controls 
Alternating Current 
Single— Phase Circuits 
Operations and Maintenance 



General Metals , 7-12 

Descriptions This collection is an introduction to 
concepts and skills in general metals. (90 objectives, 
6 items per objective) 

Contents ; Property of Metals 

Operations and Functions 
Cutting and shearing 
Filing 

Cutting Holes 
Grinding 

Forming and Bending Metals 

Metal Spinning 

Threaded Easterners 

Soldering 

Riveting 

Sheet Metal Seams 
Polishing and Buffing 
Decorating Metal 



Mechanical Drawing, 7-12 

Description : This collection is an introduction to 

concepts and skills in mechanical drawing. (85 objec- 
tives, 1 item per objective) 

Contents : Basic Drafting skills 

Beginning Lettering 
Making the Drawings 
Orthographic Projections 
Dimensioning 
Scale Drawing 
Pictorial Drawing 



Section Drawing 

Auxiliary Views 

Production Notations 

Thread Conventions and Symbols 

Assembly and Detail Drawings 

Reference Symbols 



Woodworking, 7-12 

Description : This collection introduces basic skills 

and emphasizes the processes involved in woodworking. 
(56 objectives, 6 items per objective) 

Contents: Sharpening, Adjusting, Using and Caring 

for Tools 

Reading a Working Drawing 
Stock Billing 

Laying Off Distances & Lines 
Clamping Stock 

Crosscutting and Ripping Stock 
Getting Out Rough Stock 
Removing Warp and Wind and Smoothing 
irregular Surfaces 
Planning Stock 
Chamfering 
Laying-Out Patterns 
Boring Holes 
Sawing Curves 
Making Joints 
Fastening Joints 
Sanding 
Bleaching 
Finishing 
Hanging Doors 
Mounting Locks 
Correcting Defects 
Gluing Up Stock 
Removing Finishes 



American History, 7-12 

Description : This collection emphasizes political, 

social, and economic concepts, problems and fundamental 
issues in American history from the Pre-Revolutionary 
period to modern times. (19 objectives, 6 items per 
objective) 

Contents : Historical Figures 

Historical Inferences 
Analysis of Data 
Historical Speeches 
Historical Documents 



British Colonial Acts 
Values and Goals 
Political Philosophies 
Political Protest 
Military Conflict 

Political Influence and Intervention 
Society and Culture 
Supreme Court Cases 

Constitutional Rights and Guarantees 

Legislation 

Territorial Expansion 

Geography 

United Nations 

Testing Hypotheses 



Geography . K-9 

Description ; This collection reflects major social 
science concepts in the discipline of geography. (97 
objectives, 1 item per objective) 

Contents: Geography 



Spanish, 7-12 

Description : This collection is an introduction to the 

basic concepts and skills structural, to the discipline 
of Spanish. (74 objectives, 6 items per objective) 

Contents : Understanding 

Speaking 

Reading 

Writing 



Music , K-6 

£ ^ s . cr lP'kl Q n : This collection reflects major concepts, 

fundamentals and applications in music appreciation. 
(97 objectives, 1 item per objective) 

Contents : Melody 

Harmony 

Melody and Harmony 
Rhythm 

Musical Form 
Musical Sound 
Musical Style 
Performance 
Band Instruments 
Wind Instruments 
String Instruments 
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Heal tli (nutrition) , K-6 



Description : This collection reflects the major con- 

cepts related to Nutrition — Man and His Food. ( 24 
objectives, 6 items per objective) 

Content : Daily Food Choices 

Food Processing 
Consumer Education 
Nutrients from Food 
Nutrients and Disease 
Storage - Preparation Methods 
Food in Man ' s Environment 
Table Manners 



Physical Education, K— 3 

Description ; (44 objectives, 3 items per objective) 

Content : Perceptual Motor 

Sensory Motor 
Locomotor Skills 
Non-locomotor Skills 
Balance 

Eye— Foot Skills 
Eye-Hand Skills 
Dance 



Anthropology, 4—6 

Description ; This collection contains introductory 
anthropological material utilizing some of the rnat€ j-a 
commonly studied in these grades. Different ways c 
looking at man and societies are presented. (42 ol 
jectives, 6 items per objective) 

Content : Man is a Unique Animal 

Fossil Man and Prehistory 
The Record of Culture 
The Nature of Culture 
Genetics, Evolution, and Race 
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CHAPTER 5 



QUALITY CONTROL ACTIVITIES 



Everyone supports high quality. Given enough time and enough 
resources, educational developers would prefer to produce the 
absolutely finest materials and procedures possible. But, of 
course, we never have sufficient time or resources, and it 
seems that materials scheduled for development next year are 
always needed last week. Thus , those engaged in the produc- 
tion of educational materials such as sets of instructional 
objectives and related test measures are faced with the dif- 
ficult task of deciding when the quality of their development 
efforts are of a sufficiently high quality to warrant release. 
Further, they must devise judgmental procedures, for quality 
assessment always requires judgment, and decide upon the 
requisite degree of excellence. In this section of th^ re 
P or ^- ' the activities are described which were associated with 
our efforts to assure defensible quality of the materials be- 
ing developed. 



Reviewer Based Quality Control 

In Chapter One of the report, a brief description was given 
of the increasingly systematized efforts to develop high qual- 
ity materials during the two years of the project's existence. 
In Chapter Three, the more detailed description of the dev- 
elopment procedures which were used should have revealed an 
increased sensitivity to the necessity for securing more in- 
ternal and external reviews prior to release of any products. 
The chief reason for this heightened concern about reviews 
stems from appraisals of some of the early collections after 
they were released. In spite of reasonably well coordinated 
supervision of developers, and the input/reviews of practicing 
teachers and subject matter experts , some pretty weak material 
found its way through such screens and into the early col- 
lections. Quite naturally, as one encounters defects in one's 
earlier efforts, greater care is taken with later work. in 
several of the early collections, there were too many instances, 
for example, when test items did not match objectives, when 

?£ >JeC ^ Ves • did not ™ atcl1 the descriptive categories, or when 
the objective was simply stated ambiguously. From a project 
designed to produce clear statements of objectives and related 
measuring procedures, this approximated inexcus ability . Ac- 
cordingly, as the project, continued, more and more care went 
into securing the criticism of internal and external reviewers 
who would apply stringent standards regarding quality of 
coverage, that is, the range of objectives included, and the 
quality of the materials actually contained in the collections. 
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— — Al ter n a tive C onception of Q ua lity Control 

?? em ? 1 S? ing r eviewer-based techniques for improv- 
the quality of the materials being developed, it is also 

B?iZfli e h° V1SW q ^ allty contro1 in a different fashion. 

Brief iy, by presenting to those educators who will select ob- 
jectives from the collections an array of data reqarSino 0 ?hlr 
? ro J 1 ^ s quality assessments of the objectives, the selector 

oMecX^ e r b ; e V° make a mcre enlightened se^cWon o? 

jectives which, for that situation, should be of a better 

W ° rdS ' ° ne can ™°rry about quality ap- 
praisals being made both at the point the materials arebe- 

£2 at selections art SS g ■££ ftom 

whinb ? r This IS Particularly pertinent to a project 

Tatt mtth? r to U ? ing ma ^rials designed to permit selection 
(and might be less relevant m development of less selection- 

btth^ 11 « la teriais, e.g., a textbook). w e decided to explore 
both strategies. The next several sections of this chapter 

for d f sc ^ lb ®. our efforts to devise a rationale and procedures 

effe?t-;tti eCtl0 2 f® slstance " of quality control and the 

effectiveness of these procedures when tried out extensivpl! 
m several public school settings. our extensively 



Needs Assessm ent as a Starting Point * 

One way of viewing the kinds of information an educator mirfht 
ee in making a quality assessment of possible objectives^ s 

in tSrfd 1 ^ -^ at ^ hiS is the key requirement when'one Ingag s 
or coheir 90313 !° r an instructional s ?S2m. 

tionaf nSeSs ^seLmLIf^ to ^ys ■ - educa- 

chol5Sv Sh 2dn W 9r 5 at c °ncern regarding instructional psy- 

peratfve it ij thjt ^nf been < I uite . Properly reminded how im- 
a ls that an y increased instructional sophistin*- 
l^t d i reCte f tOWard de fensible educational ou « 

employed ^Selc^tAa?^?^ “*** ^gsment has been 
tif v C be that operation which is designed to iden- 

amelioratioSf 633 ° educational deficiency most worthy of 

solution^ 3 d±sc ^ s! ? x ° n will focus on specific problems and 
JssesimeStS -r. th the con 'ihct of educational nSeds 

the outset o£ . such operations is warranted at 

niciue for i^nff educational needs assessment is a tech- 

need to be accompli she d^? Se ed Y cati ? nal objectives which most 
The conceStfoS IS h ^ a given instructional situation. 

P an educational need in this context is a 

tron v.^ h iS Sect j 0n of the report is adapted from a presents- 
in November ard‘'DL 2 ^bery e i 969 f A Tltle • Ll1 regional workshops 
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standard one. First, a desired learner outcome is identified 
Second, the learners current status with respect to that out 
come is ascertained. The difference between the current sta- 
tus and the desired status is considered to be an educational 
need. 

Once having identified a number of educational needs , the 
most difficult task of the needs assessor is to rank these 
m some way so that the educational system can be directed 
toward the satisfaction of the most important needs. 

There has been growing acceptance of the view that in order 
to adequately determine the learners ' educational needs , w«* 

a wide variety of educational outcomes, 
rather than only the customarily sought types of intellectual 
achievements. As a consequence, those working in the needs 
assessment arena are now urging educators to consider the 
identification of needs with respect to objectives in all 
three domains of learner behavior, that is, the affective, 

f^ ld Psychomotor. By cognitive, of course, we re- 
£er , to lnte l lectual types of learner outcomes. Affective 
needs pertain to attitudinal, valuing, or emotional types of 
learner outcomes . Psychomotor needs are associated with a 
learner s physical and motor skills. 

Yet, while this general approach to needs assessment has re- 
ceived substantial support from those individuals actually 
?-£. sucil activities, anyone who has attempted to im- 
p I 1 * 113 general strategy has discovered that the job is 

£»^t S S Ple * i Th !! re f r f several very thorny problems which 

e resolved, whether the needs assessment is conducted 
at a national, state, or local level. 

Tri O” *- -Identifying Educational Preferences . 

JJ^abiishing what we wish our students to become, it must 
be recognized that this operation is exclusively one of val- 

are -?° f ° rmulas which, if implemented, would ob- 
, a necessity of someone reaching judgments regarding 

what educational goals ought to be established. But even 

dJS?Ss 2 S g dS5 at ?£*•,’ ® values a ?e usually held with varying 
Mr? 1 e ff? slb i 1:Lt y' we Still encounter a number of prac» 

^ . • ^ cu determining value preferences regarding 

of the di5ir g °? 1S * -f This ±S ' ° f course - complicated because 
J S f ? r °“ ps who might wish to influence the 
1 . n desired educational outcomes. For example, 

tionSl thS “ ent * hat tllose conducting an educa- 

shonld v. „ S ass ® ssin ent identify parents as individuals who 
Wow volce ij- n the establishment of educational goals, 

m f SUCh parental preferences identified? Does a staff 

a needs assessment project interview parents and 
^ ln e ® se 2 ce ' what do y° u want your children to be 
r*? 4 the + - 1 f nd ° f thair schooling?" One suspects that re- 
sponses to this general qu .3 -ion would be given at such dif- 
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ferent levels of generality that it might be impossible to 
categorize parental preferences in a meaningful manner. Sim- 
ilarly, if academicians (or any other group) were to be polled, 
what practical methods exist for getting definitive, manage- 
able statements from those individuals? 

Although several needs assessment groups have tried to ap- 
proach this preference identification operation faithfully, 
the procedural requirements of securing an adequate set of 
preference data from potential contributors have not been re- 
solved. 

Problem Number Two: Identifying the Learners' Status . 

A problem of equal difficulty involves determining the learn- 
ers 1 current status with respect to a variety of education, 1 
outcomes. Through the years, educators have been inclined 
to use standardized achievement tests for determining the 
learners' current abilities. Recent advances in measurement 
circles, however, suggest the thorough inapprop:;iateness of 
using typical standardized tests for such assessments. The 
unsuitability of such tests tests upon a basic distinction 
between norm-referenced and criterion-referenced approaches 
to measurement. a standardized test is, generally, a norm- 
referenced test and is designed primarily to identify an in- 
dividual's stctus with respect to a norm group, that is, other 
individuals who have completed the same test. Because of the 
necessity to produce variant scores , scores which permit com- 
parisons among individuals, standardized tests are often un- 
able to represent the complete range of learner behaviors 
which we need to know about. 

Criterion-referenced tests, on the other hand, are designed 
to measure a learner's status 'with, respect to a specified 
performance standard and, as such, are more suitable for pur- 
poses of needs assessment. Unfortunately, criterion— ref er— 
enced tests do not exist in quantity. No established commer- 
cial test distributors have developed a sufficient number of 
criterion-referenced tests to be of any real utility to a 
needs assessor, arid without such tests, one cannot adequately 
measure the learners 1 current status regarding the outcomes 
in which we might be interested. 

Problem Number Three: Contrasting Preferences with Sta - 

phs ° Referring back to the general model for identifying 
educational needs, it is important to note that learners’ 
current status must be contrasted with (usually subtracted 
from) desired learner outcomes in order to determine an edu- 
cational need. Particularly because of the unsatisfactory 
methods being used to establish educational preferences and 
learner status , these comparisons are not easily produced. 

For example, what happens if a group of businessmen respond 
rather general terms to a needs assessment interviewer 
that they are looking for young men and women who can per— 
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form different kinds of clerical tasks, while actual data 
regarding learner post-high school clerical competencies are 
very specific ? Comparisons are difficult, if not meaningless. 
The procedural problems of contrasting current learner status 
with preferred learner outcomes are very real and have not 
yet received sufficient attention from those involved in edu- 
cational needs assessment. 

Collections of Instructional Objectives and Related 
^easure| . As can be seen from the three problems previously 
identified, the primary difficulties of implementing the usual 
needs assessment model are procedural. While we may have 
the wisdom to devise a general strategy, the primary techni- 
cal deficiency rests on inadequate vehicles for securing the 
right kind of data. 

One significant advance in recent months has been the estab- 
lishment of objectives-bank agencies which are attempting to 
collect large numbers of instructional objectives, stated in 
measurable terms, plus sets of devices to measure each of 
these objectives. With the existence of such collections of 
objectives and items, some procedures for dealing with each 
of the three problems identified above can be devised. A 
systematic approach to needs assessment employing these ob— 
jectives— items collections will be described in the remainder 
of this presentation. 

Problem Number One Solved . By presenting sets of meas- 
urably stated objectives to potential reference groups , that 
is, the groups of individuals whom the needs assessment staff 
wishes to involve in its survey , the needs assessor can de- 
rive a systematic set of preferences because the sets of ob- 
jectives can be presented in a relatively constant form to 
the various groups. For instance, let us suppose that we 
wish to involve representatives of (1) the community , (2) the 

— themselves , and ( 3 ) educ ator s in establishing the 
desired goals of an educational system. A set of objectives 
from available collections , each perhaps accompanied by a 
sample measurement item to more clearly communicate the nature 
of the objective » would t>e presented to representatives of 
each of these three groups. It might be necessary to modify 
the language of the objectives somewhat for the different 
9 rou P s • f < p r if may not be realistic to expect young learners 
and certain community representatives to understand the tech- 
nical language which would be comprehensible to educators. 
However, even with such modifications, the basic set of ob- 
jectives would remain the same. 

We could ask these groups to appraise the objectives in a 
variety of ways. A very simple approach, for example, might 
ask the representative individuals to rate or rank each of the 
objectives for purposes of its possible inclusion in the school 
curriculum. Ratings might be supplied according to the degree 
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of importance the individual attached to the objective. 

These ratings, for example, supplied on a five point scale, 
could be used in developing average estimates of the import- 
ance attached to each objective by the particular group in- 
volved. The preferences of the several groups could be ar- 
rayed in a relatively simple manner which would permit com- 
parisons among the average ratings for each objective and, 
course , the identification of those objectives considered 
most important by all groups. Similarly, objectives considered 
unimportant by all three groups would, once identified, un- 
doubtedly be eliminated from further consideration. In in- 
stances where there was significant disagreement among the 
participant groups, further exploration among group represen- 
tatives might reveal the reasons for such disagreement and, 
possibly, a way to reconcile the disparate ratings. On the 
other hand, in such instances of disagreement, it is perfectly 
reasonable to attach more weight to the preferences of one 
group than another. This is a philosophic issue which would 

c early have to be faced by those conducting the needs assess- 
ment . 



P roblern Number Two Solved . As indicated before, the use 
of norm-referenced tests for the assessment of current learner 
competencies is impermissable . However, the existence of 
pools of test measures for the objectives which are being used 
makes possible the ready preparation of criterion-referenced 
tests. a set of such items, randomly drawn from the available 
pools, could be administered to the learners and data could 
be secured regarding the degree to which the learners were 
able to master the objectives. These measuring items are not 
necessarily designed to produce variance among learners. They 
are simply designed to bs congruent with the objective and, 
as such, represent the most adequate reflection of the objec- 
tive's attainment. The avoidance of standardized tests and 
the use of such criterion— referenced test items (for each ob- 
jective identified as desirable, on the basis of preference 
data) will yield a far more sensitive reflection of current 
learner status regarding’ each, objective. 



g rcbiem Number Three S olved . Since both the objectives 
and the test measures employed are drawn from the same agency, 
comparisons of preference and current status data are rendered 
far more simple. We would only have to calculate the average 

estimates for eac h of the involved groups, then 
identify the percent of students mastering the objectives. 
These comparisons could be presented in summary form such as 
seen in Table 4. 



It is now necessary, of course, to make those difficult deci- 

Regarding which objectives, among competing alternatives, 
reflect the most important educational needs. While we do 
not yet have a simple computer program which could take such 
data and pump out a precise identification of the most cru- 



cial student needs, a careful appraisal of such information 
should yield far more enlightened choices among alternative 
needs than by using currently extant schemes. 





PREFERENCE DATA 
Mean Ratings 




PERFORMANCE DATA 
Pre— instruction 


Objective 


Community 


Learners 


Educators 


%Masterinq 


#1 


4.3 


4.6 


4.4 


14 


#2 


2.1 


2.5 


3.1 


68 


#3 


1. 2 


1.4 


1.5 


20 


#4 


4. 7 


2.1 


4.0 


31 


#5 
etc . 


3.1 


4.6 


2.0 


84 



Table 4 - A Hypothetical Comparison of Preference and Per- 
formance Data . 



person and Item Samp ling. in using the procedure recom- 
mended here, it is important to use sampling procedures judi- 
ciously throughout, both with respect to sampling the people 
involved as well as the items to wh.i th~ v are responding. 

For purposes of economy, we would dly w i s i! to select 

only a sample of individuals fre ral clienteles in- 

vo ve . For in stance , if we wif .se teacher groups ii. 

a state-wide assessment, then we Oaynt to employ sampling 

P ^°£ e ?" re f S ? that we do not burden all teachers in the state 
with the task of rating objectives. Similarly, samples of 
students would need to be chosen for testing purposes. 

While the value of such person sampling o.s generally recog- 
nized, the utility of item sampling has not been comprehended 
Z* ? e f ds assessment personnel. item sampling permits one 

administer different items to different people in order to 
T nn 31 ? 5 group estimate. In other words, if I am testing 
100 students and wish to know how the 100 students perform 
on four objectives , I might constitute a test of four separate 
parts, each part reflecting one separate objective, then ran- 
domly administer each of the different sections to only 25 
stutJents. Since a sizable portion of the class would be re- 
sponding to each test item, I could secure a perfectly ade- 

those items atl ° n ° f the degrse to which students could master 



Similarly, we need not subject parents to the necessity of 
rating hundreds of objectives. We could, instead, put to- 
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gether a variety of different short sets of objectives and 
let randomly selected parents rate anywhere from 10 to 20 
objectives, thereby taking no more than a few minutes of each 
parent s time. The use of person sampling and item sampling 
procedures is requisite in the economic implementation of 
this approach to needs assessment. 

A _S elf -Corr ecting Quality Control System . While not all 
of the problems to be faced in implementing this approach to 
educational needs assessment systems have been identified, 
for it has not been used on a wide-scale basis, certainly’ 
tne strategy is sufficiently manageable to permit improve— 
ment as it is employed. Through the assembly of collections 
of instructional objectives and related criterion measures, 
coupled with the efforts of those who would systematically 
determine educational needs , we will surely increase the 
quality of our educational needs assessment operations so 
that we can identify the high quality educational objectives 
we really ought to be pursuing. 



— t he Selection— Assistance Strategy 

As the foregoing approach to securing preference ratings as 
se ection— assistance data became more formalized, several 
sm ^ -scale field trials were undertaken, some involving only 
a few students, parents, or educators. A somewhat more ex- 
tensive field test was conducted in January, 1971 to assess 
the adequacy of the procedures developed at that point. This 
next section of the report describes that investigation.* 



The study focused on a collection of 19 U. S. history objec- 
tives developed as part of the project. Each objective was 
rated by four different groups. Average ratings from four 

° f ? e °P le were thus secured and, subsequently, 
rankings of the various objectives in order to provide some 
gross estimates of the degree to which each objective was 
preferred by a different type of rater, that is, the quality 
assessments made by each giroup- 



^ Subjects . The four rater groups involved were students, 

71 tenth grade pupils; teachers , 17 U. S. history teachers; 

: ?? P? rents ? f high school youngsters; and futurists, 

future t0 rata of their 



The teachers, students , and parents were drawn from a middle 
class suburban school district in Southern California. The 
students came from three tenth grade classes, while the 



_ * Adapted from a presentation by 
meeting of the California Educational 
April 29-30, 1971. 



the writer at the annual 
Research Association, 
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teachers, each instructing one U. S. history class, were all 
employees of the same school district. Seven of the parents 
were parents of students involved in the student sample, 
while eighteen were parents of students in a neighboring 
high school. 

The futurists consisted of four advanced graduate students 
(economics, history, education, English), three professors 
(economics, English, history), two professionals (a pharma- 
cist and a lawyer) , and one housewife. Each individual in 
the f uturist group had either already read Alvin Tof f ler ’ s 
Future Shock, which deals with the importance of preparing 
for a future society or were obliged to read a 40-page ex- 
cerpt from that volume. They were instructed to project them- 
selves forward to the year 1985 and rate the history objec- 
tives according to their suitability for that future period. 
The purpose of utilizing a futurist group was , of course , to 
test the viability of a scheme such as Toff ler proposes in 
which "Counsels of the Future" serve to guide educational 
systems in such a fashion as they will be more appropriate 
for a more rapidly changing society. 

Procedure . The four rater groups were provided with a 
set of the nineteen objectives and a rating form which asked 
them to rate each objective on a five-point scale in terms 
of its importance that students be able to achieve the objec- 
tive. A five-point rating reflected an objective which was 
deemed extremely important, while a rating of one indicated 
an extremely unimportant objective. Copies of the instruc- 
^-i° ns given to each rater group and the nineteen objectives 
involved are supplied in Appendix a. 

Results . In Table 5 , the mean ratings for each objec— 
x 'i ve are presented for the four rater groups, as well as a 
inking by a group of each objective based on these mean 
-mgs. 



It can be seen that, as expected, there was considerable 
variation among the mean ratings supplied by the four dif- 
ferent groups. The average mean rating of the students was 
2.96, for the teachers it was 3.25, for the parents it was 
3.27, and for the futurists it was 3.31. The degree to which 
the rankings of the objectives by the four groups tended to 
coincide is revealed by the rank order intercorrelations 
among the four sets of rankings. These relationships are 
presented in Table 6 , where it can be seen that very strong 
relationships were present among rankings supplied by stu- 
dents, teachers, and futurists, while the parent group pro- 
duced ratings which were at some variance with the other 
three groups . 

A Procedu ral Question . There are a number of procedural 
questions, some minor and some major, which must be examined 
in using an approach such as this. For example, to what ex— 
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Table 5 



Ranked Mean Ratings of Nineteen History Objectives 
by Students, Teachers, Parents, and Futurists. 



Objective 


students 
n = 71 
% Rank 


Teachers 
n = 17 
X Rank 


Parents 
n = 25 
X Rank 


Futurists 
n = 10 

V ■Rj=,r->lr 




A 


3.09 


8 


3.29 


10 


3.72 


4* 


3. 00 


12 




B 


2.84 


13 


3.12 


12 


3.17 


12 


3.90 


6 




C 


2.47 


18 


2.65 


17* 


3.72 


4* 


2 „ 50 


16* 




D 


3.06 


9* 


3.18 


11 


3.12 


14 


2.90 


13 




E 


3.18 


4 


4.12 


1 


3.42 


9* 


4.60 


2 




F 


2.90 


12 


2.88 


15 


3.00 


15 


2.60 


15 




G 


3.26 


3 


3.47 


6* 


3.88 


2 


3.60 


8 




H 


2.83 


15 


3.35 


9 


3.58 


6 


3,20 


10 




I 


3.06 


9* 


3.00 


14 


3.56 


7* 


3 . 30 


9 




J 


3.48 


i 


3.88 


3 


3.56 


7* 


L. IQ 


4 




K 


3.17 


5 


3.53 


5 


3.84 


3 


^.00 


5 




L 


3.13 


7 


3.47 


6* 


3.28 


11 


3.80 


7 




M 


2.14 


19 


2.29 


19 


2.56 


16 


2. 50 


16* 




N 


3.38 


2 


3.59 


4 


4.16 


1 


4. 20 






O 


2.85 


13 


3.06 


13 


3.42 


9* 


2.70 


14 




P 


2.67 


16 


2.82 


16 


2.43 


18 


2. 20 


18 




-Q 


2.61 


17 


2.65 


17* 


2.00 


19 


2.00 


19 




R 


3.01 


11 


3.41 


8 


3.16 


13 


3.10 


11 




S 


3.16 


6 


4.06 


2 


2.55 


17 


4. 70 


i 





* indicates tied ranking 





Table 6 . 
rankings 



Kank order intercorrelations among objectives 
by students/ teachers, parents, and futurists. 



Students 



Teachers Parents Futurists 



. P9 .48 



.83 



Teachers 



.41 



.91 



Parents 



.42 



Significance levels: p 



05 



R s =.46; 



P 



.01, R s =.6Q 



are . av ~ ra ^ e ratings (and subsequent rankings) of obiee- 
tives satisfactory or should alternative schemes e.a. Da irpH 
comparison approaches be employed? How will variations in 
the instructions to tne raters modify the differential pre- 

cal^ein ^hSt^t ** Y arious groups? In a more practi- 
cal TS * ' h d f ta gathering procedures will prove economi- 
cal m securing large quantities of such preference data? 

During the course of the project, a number of procedural 

Fo^eximS? WerS tested Regarding these kinds of questions. 

For example, one investigation attempted to verify the ef- 
ficacy of several different procedures for securing parental 

proW?'v.ent‘ Variatiun ’ representatives ? rSS Se 

fa to , the classroom and administered the survevs to 

and return f tUdents to carry ?hem to^ar^ts 

3 mall"Scslp t ftoi 2 f ^ Sr ? ar f, I ! ts had completed the forms. In a 
return The d trial, this procedure yielded a 36 per cent 
return. The second procedure involved the teachers' receiving 

in mail (no project personnel present 9 

to n h ^ S ? h ° 0l 2 Wlth ^ arental rating forms once more^arried 
oL? 3 e ? tS and retu ™ d by students! In this case a 68 oer 

ministered^^mail 6 ? * *- A t £ ird Procedure found materials ad- 
home to teachers but students asked to provide 

ratine add *esses on a response form. Subsequently, 

rating forms were mailed directly to parents with instruc 

le^ni^ e r vI'eSJ hSm £ the « 1 “™ “SS 

was essential!^ P ® r C ® nt return ‘ The fourth procedure 

turned the rat^n -f s 5 me . as third except that parents re- 

procedure vielderf ^g? 3 malil not ^ their children. This 
procedure yielded a 91 per cent return. 

of e enterDris^ t ^ Vene ^i consideration involved in these kinds 
be diffi?£itiL ar ? 31 “"resolved. There would obviously 
^„ d H5 fibres, for example, m having students supply their 
parents home addresses, for in certain school districts 
this would undoubtedly be viewed as an invasion ^ p^vacy. 
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This brief account is supplied only to indicate the myriad 
procedural questions one faces using this approach. Most of 
the questions are amenable to empirical solutions but, of 
course, take a considerable amount of energy. 

Even though the resources requisite to study the problems of 
goal determination via preferential ratings of extant objec- 
tives are considerable, the possibility of producing an .im- 
provable technology for curriculum workers seems a sufficiently 
alluring prize that the energy expenditures may well be war- 
ranted. 



A Major Field Trial * 

In May, 1971, the project staff, in cooperation with the 
Culver City Unified Schoo.l District (Culver City, California), 
conducted a rating study asking three subject groups to rate 
certain objectives. The purpose of the rating study was two- 
fold. First, it was designed to test procedures developed 
during the project for the obtaining of such ratings. Second, 
it was designed to obtain ratings of the specific objectives 
used, information which would be used in the revision of the 
collections from which the objectives were chosen. 

Objectives Used . The objectives used in the rating 
study were those which formed the Study Skills sections of 
the Reading collections distributed by the Instructional Ob- 
jectives Exchange. Thirty objectives were taken from the 
K— 3 Reading collection, fifty-two objectives from the 4-6 
Reading collection, and twenty-five from the 7—12 Reading 
collection. 

Subjects * The survey was conducted using teachers and 
classrooms from six elementary schools , one junior high 
school, and one senior high school. Fifty— six classrooms 
formed the basis of the survey. 

The three rater groups involved were students , 575 at the 
4—6 grade levels and 323 at the 7—12 grade levels; parents , 

186 at the K— 3 grade levels, 364 at the 4—6 grade levels, 
and 104 at the 7-12 grade levels; and teachers, 15 at the 
K-3 grade levels, 23 at the 4-6 grade levels, and 7 at the 
7—12 grade levels. 

The survey encompassed subjects from all income levels , high 
through lower, and different socio-economic backgrounds. Tlie 
teachers, at the junior and senior high school levels (grades 
7—12) were all instructing at least one English or reading 
class. All of the parents were parents of students involved 
in the student sample. 



*This section of the report was prepared by Hal S. Malkin, 
who also coordinated the field trial. 
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Procedure . A complete description of the procedure and 
documents used can be found in Appendix B. Presented here 
is a brief summary of the procedure used to aid in the under- 
standing of the results. 

Survey kits were distributed to the fifty-six classrooms 
which were chosen to participate in the survey. Sixteen 
classrooms received the K-3 objective sets, twenty-eight class- 
rooms received the 4-6 objective sets, and twelve classrooms 
received the 7-12 objective sets. Half of the classrooms 
received objective sets which contained a sample test item 
included with each objective, while the other half received 
objective sets which were without sample test items. Because 
previous field trials had indicated that the maximum number 
of objectives students and parents were willing to rate be- 
fore boredom or fatigue set in was approximately ten objec- 
tives, the objective sets were randomly broken down as fol- 
lows*: 

1* The thirty K-3 grade level objectives into three 

sets of ten objectives each, each set labeled either 
A, B, or C. 

2. The fifty— two 4—6 grade level objectives into two 
sets of eight objectives and four sets of nine ob- 
jectives labeled D, E, F, G, H, and tf. 

3. The twenty-five 7-12 grade level objectives into 
two sets of eight objectives and one set of nine ob- 
jectives, labeled M, N, and P. 

The survey sets were distributed in the classroom by the 
teacher. The students then completed their r»»— t of th : tou .jc- 
vey and were instructed to take hr ^ set an< j have 

the parents complete their part. The parent s response form 
V/ as then returned to the school. The teacher was also re- 
quested to rate the objectives. 

Each objective was to be rated on a five— point scale in terms 
i desirability of being included in classroom instru : — 
tion. A five— point rating reflected an objective which defi- 
nitely should, be included in the instruction, while a rating 
one indicated an objective which should definitely not be 
included in classroom instruction. 

Results . Of the 53 classrooms which received survey 
kits, completed response forms were received from 51. Six 
of those received had to be rejected either because they 
were incomplete or had be- i improperly completed. Thus, this 
yielded 45 sots from which to obtain the desired information. 



*All objective sets are included in 
purposes of illustration, the test items 
included in Appendix D. 
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Appendix C. For 
used with Set A are 



Parent response forms numbered 52 per cent of the total num- 
ber of student response forms received at the 4-12 grade levels. 
(Students in the K-3 grade levels did not receive response 
forms; only the parents and teachers were requested to rate 
the objectives. ) Classroom returns show a range from 14 per 
cent to 100 per cent in terms of parent responses to student 
responses . 

In Table 7 , the mean ratings for each objective in the K-3 
objective set are presented for the two rater groups. Also 
presented is a ranking by group of each objective based on 
these mean ratings. At the bottom of each column is presented 
the average mean rating supplied by each group. 

In three of the six groups there was considerable variation 
among the mean ratings supplied by the two groups , although 
both groups tended to rate the objectives high on the scale. 

The degree to which the rankings of the objectives by the 
two groups tended to coincide is revealed by the rank order 
correlations between the two sets of rankings. These rela- 
tionships are presented in Table 8 where it can be seen that 
in only one case was a very strong relationship present among 
rankings supplied by the parents and teachers. 

Table 9 presents the mean ratings for each objective in the 
4-6 objective set. The entire set was divided into six 
smaller sets / and the data are presented in terms of these 
smaller sets. In this instance, data are included for students 
as well as for parents and teachers. 

There w<,.,^ cons lerable variation among the mean ratings sup- 
plied by the three different groups . The average mean rat- 
ing of the different groups, shown at the bottom of the aver- 
age rating column, ranged in one case (Set H without sample 
test items) from 2.79 for students to 4.10 for parents and 
4.08 for teachers. Table 10 presents the rank order intercor- 
relations among these three groups. 

Table 11 presents the mean rating for each objective in the 
7—12 grade level objective set. Again we find a large vari- 
ation in the average mean ratings among the three groups , 
and again the student ratings tended to have a greater variance 
with parent and teacher ratings than did parent and teacher 
ratings with each other. Table 12 presents the rank order 
intercorrelations among these three groups. It can be seen 
that in four of the six cases there was a significant cor- 
relation between the student and parent rankings and in three 
of the six cases a significant correlation between the parent 
and teacher rankings. But in only one case was there a sig- 
nificant correlation between student and teacher rankings. 

As stated earlier, one of the main purposes of the field trial 
was to test procedures developed during the project for the 
obtaining of user preference data. 
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Table 7 



Ranked Mean Ratings of 30 Study Skills Objectives 
by Parents and Teachers (K-3 Grade Levels). 



GROUP A 



Objective 
(With Sample 
Test Item) 


Parents 
n = 25 
52 Rank 


Teachers 
n = 7 
X Rank 




Al 


3.88 


6 


4.00 


6* 




A2 


4.60 


1 


4.14 


5 




A3 


4. 24 


4 


4.71 


2 




A4 


3.56 


10 


4. 29 


4 




A5 


3.84 


7 


4.57 


3* 




A6 


3.76 


8 


3.71 


8 




A7 


4. 29 


3 


4.86 


1 




A8 


3.68 


9 


4.00 


6* 




A9 


4. 16 


5 


3.17 


9 




A10 


4.46 


2 


4.57 


3* 






4.05 




4. 20 







Objective 
(Without Sample 
Test Items) 


Parents 
n = 32 
X Rank 


Teachers 
n = 7 
X Rank 


Al 


2.92 


10 


3.43 


10 


A2 


4.63 


1 


3.86 


8 


A3 


3. 66 


8 


5.00 


1 


A4 


3. 78 


7 


4.23 


5 


A5 


3.88 


4 


4.86 


2* 


A6 


3. 84 


6 


4.57 


4 


A7 


4.61 


2 


4.86 


2* 


A8 


3.87 


5 


4.00 


7 


A9 


3.47 


9 


3.57 


9 


A10 


4.50 


3 


4.14 


6 




3.92 




4. 25 





* indicates tied ranking 
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GROUP B 



Objective 
(With Sample 
Test Item) 


Parents 
n = 27 
X Rank 


Teachers 
n = 7 
X Rank 


Bl 


4. 37 


7 


4.43 


3* 


B2 


4. 31 


8 


3.86 


6 


B3 


4. 17 


10 


4.00 


6 * 


B4 


4.41 


6 


4.00 


6 * 


B5 


4. 30 


9 


3.43 


10 


B 6 


4. 56 


3 


4. 86 


1 


B7 


4. 70 


1 * 


3.71 


9 


B 8 


4. 70 


1 * 


4.57 


2 


B9 


4.48 


4 


4.43 


3* 


BIO 


4.43 


5 


4.43 


3* 



4.44 4.17 



Obnective Parents Teachers 

(Without- Sample n = 32 n = 7 

— Test Item) X Ranh X Rank 



B1 

B2 

B3 

B4 

B5 

B 6 

B7 

B 8 

B9 

BIO 



*indicates tied ranking 



3.93 


9 


3.86 


8 


4.68 


1 


4. 71 


1 * 


3,97 


8 


4. 23 


6 * 


3.83 


10 


4.71 


1 * 


4.09 


7 


3.57 


9 


4. 28 


4 


4. 57 


4 * 


4.23 


5 


3.43 


10 


4.45 


3 


4. 57 


4 * 


4. 55 


2 


4. 71 


i* 


4.14 


6 


4.23 


6 * 



4.22 4.26 
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GROUP C 



Objective Parents Teachers 

(With Sample n = 19 n = 7 

Test Item) X Rank X Rank 



Cl 


4. 58 


C 2 


4.68 


C3 


4.68 


C4 


4.16 


C5 


4.47 


C 6 


4.21 


C7 


4.47 


C 8 


4.22 


CS 


4.00 


CIO 


4.16 



4, 36 



3 


4.14 


8 


1 * 


4.43 


3* 


1 * 


4. 71 


1 * 


8 * 


4. 29 


6 * 


4 * 


4.43 


3* 


7 


4.29 


6 * 


4 * 


3.57 


9 * 


6 


3.57 


9 * 


0 


4. 71 


1 * 


8 * 


4.43 


3* 



4. 26 



Objective Parents Teachers 

(Without Sample n = 22 n = 7 

— Test Item) X Rank X Rank 



Cl 


4.57 


C2 


4.50 


C3 


4.58 


C4 


4.28 


C5 


4.58 


C 6 


3.42 


C7 


4.58 


C 8 


3.47 


C9 


4.50 


CIO 


4.17 



4.27 



4 


4.57 


5 


5* 


4. CO 


9 


1 * 


4.86 


2 * 


7 


3.86 


10 


1 * 


4.86 


2 * 


10 


4.23 


7 


1 * 


5.00 


1 


9 


4.14 


8 


5* 


4. 71 


4 


8 


4.43 


6 




4.47 





* indicates tied ranking 
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Table 8. Rank Order Intercorrelations Among Objectives by- 
Parents and Teachers (K-3 Grade Levels). 



Parent /Teacher 



Objective Set A 



With sample test items 
Without sample test items 


. 51 
. 26 


Objective Set B 




With sample test items 
Without sample test items 


.53 
. 35 


Objective Set C 




With sample test items 
Without sample test items 


-.22 
. 75* 



* significant at p < .05 level 



Cost effectiveness considerations combined with results ob- 
tained in the small-scale field trials described earlier re- 
sulted in the distribution system chosen for the large-scale 
-ield trial, (See Appendix B for a description of the various 
systems tested. ) Using the school district ' s mail distribu- 
"“ on system to get the materials to the classroom and having 
the parental rating forms carried to parents and returned by 
students proved to yield the greatest return per dollar cost. 
This system was used in the large scale field trial. 

Of the forms transmitted to parents of 4-12 grade children, 

52 per cent were returned. For the parents of K-3 children, 

61 per cent were returned. The number of returns at all levels 
K-12 were judged more than adequate to provide the desired 

information. Thus, this aspec t of the procedure was considered 
satisfactory. 

A second major procedura. question concerned the insertion of 
the sample test item with each objective. Half of the sur- 
vey kits which were distributed contained only objectives 
for the subject groups to rate, while half of the survey kits 
contained objectives plus a sample test item. Table 13 pre- 
sents the rank order intercorrelations among objective sets 
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Table 9 



Ranked Mean Ratings of Fifty-two Study Skills Ob- 
jectives by Students, Teachers, and Parents (4-12 
Grade Levels). 



GROUP D 



Objective 
(With Sample 
Test Item) 


Students 
n = 34 
X Rank 


Parents 
n = 28 
X Rank 


Teachers 
n = 4 
X Rank 


Dl 


3,58 


5 


4.25 


6 * 


3.50 


7 


D2 


3.79 


1 


4.82 


1 


5.00 


1 


D3 


3.65 


3 


4.25 


6 * 


4. 25 


4 * 


D4 


3.60 


4 


4.33 


4 


4. 25 


4* 


D5 


3.72 


2 


4.64 


2 


4.00 


6 


D 6 


3.38 


8 


4.52 


3 


4. 75 


2 * 


D7 


3.39 


6 * 


3.64 


8 


3.25 


8 


D 8 


3. 39 


6 * 


4.26 


5 


4.75 


2 * 




3.56 




4.34 




4. 22 





Objective 


Students 


Parents 


Teachers 


(Without Sample 


n = 47 


n — 30 


n = 6 


Test Item) 


X Rank 


X Rank 


X Rank 



Dl 


3.14 


D2 


3 . 74 


D3 


3.46 


D4 


3.52 


D5 


3.24 


D 6 


4.04 


D7 


3.00 


D 8 


3. 33 



3.43 



*indicates tied ranking 

O 

ERIC 



7 


4.34 


5 


3.67 


4* 


2 


4.63 


1 


3.57 


7* 


4 


4.43 


3* 


3.67 


7 * 


3 


4.23 


7 


3.80 


3 


6 


4. 26 


6 


4 • 00 


2 


1 


3.93 


8 


5.00 


1 


8 


4.46 


2 


3.57 


7* 


5 


4.43 


3* 


3.67 


4* 




4.34 




3.87 
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1 S) 


SO 









Objective Students Parents Teachers 

(With Sample n = 35 n = 25 n = 4 

Test Item) X Rank X Ran k X Rank 



E1. 


3.49 


2 


4.40 


2 


5.00 


1* 


E2 


3.42 


4 


4.36 


3 


5.00 


1* 


E3 


3.97 


1 


4.12 


4 


4.75 


4 


E4 


3.43 


3 


4.08 


5 


5.00 


1* 


E5 


3.34 


5 


3.92 


6 


4.50 


5* 


E6 


3.03 


8 


3.56 


9 


3.00 


9 


E7 


2.87 


9 


3.88 


7 


4.25 


7 


E8 


3.18 


7 


3.80 


8 


3.50 


8 


E9 


3.21 


6 


4.42 


1 


4. 50 


5* 




3.33 




4.06 




4. 39 





Objective Students Parents Teachers 

(Without Sample n = 50 n=28 n=6 

Test Item) X Rank X Rank X Rank 



El 


3.70 


2 


4.11 


2 


4.33 


1 * 


E2 


3.47 


3 


4.82 


1 


4.33 


1 * 


E3 


3.20 


6 


3.90 


4 


3.83 


6 


E4 


3.81 


1 


3.93 


3 


4.33 


1 * 


E5 


3.35 


4 


3.75 


6 


4.00 


4* 


E6 


3.13 


*1 

/ 


3.62 


8 


4.00 


4 * 


E7 


3.02 


9 


3.65 


7 


3.50 


7 + 


E8 


3.10 


8 


3.50 


9 


3.17 


9 


E9 


3.29 


5 


3.86 


5 


3.50 


7* 




3.34 




3.90 




3.89 





* indicates tied ranking 
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GROUP F 



Objective Students Parents " Teachers 

(With Sample n = 38 n=24 n =4 

Test Item) X Rank X Rank X Rank 



Fl 


3.56 


8 


F2 


4.09 


2 


F3 


3.94 


3 


F4 


4.19 


1 


F5 


3.83 


4 


F 6 


3.42 


9 


F7 


3.82 


5 


F 8 


3.56 


7 


F9 


3.64 


6 



3.78 



4.00 


7 


4.00 


6 * 


4.71 


1 


-5.00 


1 


3. 58 


8 


4. 50 


4 * 


4. 22 


5 


4. 75 


2 * 


4. 50 


2 


3.00 


9 


3. 21 


9 


3.25 


8 


4.43 


4 


4.50 


4* 


4. 48 




4.75 


2 * 


4.12 


S 


4.00 


6 * 



4.14 4.19 



Objective Students Parents 

(Without Sample n = 54 n — 34 

Test Item) X Rank X Rank 



Teachers 
n = 6 
X Rank 



Pi 

F2 

F3 

F4 

F5 

F 6 

F7 

F8 

F9 



3.22 

4.02 

3.13 
3.75 

3.14 
3.57 
3.20 
3.53 
3.29 

3.43 



6 

1 

9 

2 

8 

3 
7 

4 

5 



4.03 
4.67 
3.53 
4. 24 
4.06 
3.12 
4.33 
3.75 
4.00 

3.97 



5 
1 
8 

3 

4 
9 
2 
7 

6 



3.50 
4.33 
4. 00 
3.57 
4.00 
3.17 
3.83 
3.17 
3.50 

3.67 



6 * 

1 

2 * 

5 

2 * 

8 * 

4 

8 * 

6 * 



* 1 



indicates tied ranking 



O 

ERIC 
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GROUP G 



Objective Students Parents Teachers 

(With Sample n = 36 n = 33 n 6 

Test Item) X Rank X Rank X Rank 



Gl 


3.83 


4 


4.42 


6 




6 


G2 


3.82 


5 


4.55 




4. • * 


1* 


G3 


3.53 


7 


4.68 


1 


2.3 




G4 


4.03 


2 


4.48 


4 


4.0, 


4* 


G5 


3.92 


3 


4.52 


3 


4.67 


1* 


G'o 


3.63 


6 


4. 21 


8 


3.33 


3 


G 7 


3.23 


9 


3.91 


9 


3.50 


1 


G8 


4. 29 


1 


4.33 


7 


4. 50 


3 


G9 


3.35 


8 


4.45 


5 


4.00 


i* 




3.74 




4. 39 




3.85 





Objective 
(Without Sample 



Students 
n = 35 



Parents 
n = 26 



Teachers 
n = 4 



Item) 


X 


Rank 


X 


Rank 


X 


Rank 


Gl 


3.18 


4 


4.40 


1* 


4.25 


4* 


G2 


2.85 


9 


4. 35 


3 


4. 50 


2* 


G3 


3.29 


2 


4.16 


4 


3.25 


8 


G4 


3.09 


5 


4.08 


5 


3.50 


7 


G5 


2.94 


8 


3.96 


6 


4.50 


2* 


G6 


3.24 


3 


3.79 


7 


4.00 


6 


G7 


3.03 


6 


3.67 


8 


3.00 


9 


G8 


3.45 


1 


4.40 


1* 


4. 75 


1 


G9 


3.00 


7 


3.56 


9 


4. 25 


4* 




3.12 




4.04 




4.00 





* indicates tied ranking 



3 

ERIC 
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GROUP H 



Objective 


Students 


Parents 


Teachers 


(With Sample 


n = 


32 


n = 


29 


n = 


5 


Test Item) 


X 


Rank 


X 


Rank 


X 


Rank 


HI 


4.09 


1* 


4.45 


2 


3.67 


3* 


H2 


4.09 


1* 


4.48 


1 


3.83 


2 


H3 


3.40 


7 


3.34 


8 


2.67 


8 


H4 


3.93 


4 


4.14 


4 


3.67 


3* 


H5 


3.77 


6 


3.69 


6* 


3.57 


5* 


H6 


4. 00 


3 


4.38 


3 


3.57 


5* 


H7 


3.67 


5 


4.07 


5 


4.50 


1 


H8 


3.03 


8 


3.69 


6* 


3.57 


5* 




3.75 




4.03 




3.63 




Objective 


Students 


Parents 


Teachers 


Zithout Sample 


n = 


38 


n — 


22 


n = 


5 


Test Item) 


X 


Rank 


X 


Rank 


X 


Rank 



Hi 


2.95 


H2 


3.37 


H3 


2.41 


H4 


2.86 


H5 


2.81 


H6 


2.65 


H7 


2.65 


H8 


2.58 



2.79 



*indicates tied ranking 



ERIC 



2 


4.43 


1 


3.80 


7 


1 


4.14 


5 


4.40 


2* 


8 


3.41 


8 


2.80 


8 


3 


3.91 


7 


4.60 


1 


4 


4.32 


3* 


4. 20 


4* 


5* 


4.32 


3* 


4.40 


2* 


5* 


4.33 


2 


4. 20 


4 * 


7 


3.95 


6 


4. 20 


4 * 




4.10 




4.08 
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GROUP J 



Objective Students ” Parents Teachers 

(With Sample n = 32 n = 24 rx = 6 

_Test Item) X Rank X Rank x Rank 



J1 


3.70 


6 


J2 


3.34 


9 


J3 


4.06 


1 


«J4 


3.97 


3* 


J5 


4.03 


2 


J6 


3.97 


3* 


J7 


3.63 


7 


J8 


3.93 


5 


J9 


3.53 


8 



3.80 



4.00 


5 


4.50 


1* 


4.08 


4 


2.50 


9 


4. 38 


3 


4. 50 


1* 


3.77 


7* 


3.00 


7* 


3.78 


6 


3.57 


5 


4.43 


2 


3.17 


6 


3.77 


7* 


3.00 


7* 


4.77 


1 


4.00 


4 


3.68 


9 


4.33 


3 



4.07 3.62 



Objective Students Parents 

(Without Sample n = 41 n = 31 

T est Item) 5? Rank x Rank 



Teachers 
n = 5 
X Rank 



J1 


3.59 


2 


J2 


2.75 


8 


J3 


3.56 


3 


J4 


2.63 


9 


<J5 


3.13 


4 


J6 


2 = 92 


/ 


J7 


3.08 


5* 


J8 


3.08 


5* 


J9 


3.91 


1 



3.18 



4.55 


2 


4.80 


1 


3.68 


8 


2.20 


9 


4.71 


1 


4.40 


2* 


4.39 


3 


3. 20 


8 


3.55 


9 


4. 20 


4 


3.93 


6 


3.40 


7 


3.90 


7 


3.80 


5 


3.97 


5 


3.60 


6 


4.31 


4 


4.40 


2* 



4.11 3.78 



* indicates tied ranking 
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Ta >le J 



-Kahk. Order tercorrelations Among Objectives by 
Students, Parents, and Teachers (4-6 Grade Levels). 



Objective Set D 

With sample test items 
Without sample test items 



Objective Set E 

With sample test items 
Without sample test items 



Objective Set F 

With sample test items 
Without sample test items 



Objective Set G 

With sample test items 
Without sample test items 

Objective Set H 

With sample test items 
Without sample test items 

Objective Set J 

With sample test item 
Without sample test item 



Student/ 

Parent 



.48 

-.29 



.62 

.82* 



.47 
. 25 



.15 

.38 



. 95* 
.40 



.32 

.36 



Student/ 

Teacher 



.15 
. 30 



.83* 

.85* 



.55 

-.11 



. 56 
-.14 



.59 

.43 



.35 

.97* 



Parent/ 

Teacher 



.67 

-.90 



.75* 

.77* 



.42 
. 63 



. 28 
.47 



.67 

.09 



.15 

.50 



*signif leant at p < .05 level 



O 

ERIC 
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Table 11. Ranked Mean Ratings of Twenty-five Study Skills 
Objectives by Students, Parents, and Teachers 
(7-12 Grade Levels). 



GROUP M 



Objective 
( Wi th S ample 
Test Item) 


Students 
n = 37 
X Rank 


Parents 
n = 16 
X Rank 


Teachers 
n = 4 

Rank 


Ml 


3.35 


2 


3.93 


7* 


4.75 


1* 


M2 


3.00 


6 


3.93 


7* 


3.50 


7* 


M3 


3.03 


5 


4.13 


3* 


4.00 


6 


M4 


3.22 


4 


4.00 


5 


4.50 


3 


M5 


2.89 


8 


3.94 


6 


4. 25 


4* 


M6 


2.97 


7 


4. 13 


3* 


4. 75 


1* 


M7 


3.33 


3 


4. 44 


2 


3.50 


7* 


M8 


3.38 


1 


4.63 


1 


4. 25 


4* 




3.14 




4.14 




4.19 





Objective 


Students 


Parents 


Teachers 


(Without Sample 


n = 27 


n = 13 


n = 3 


Test Item) 


X Rank 


X Rank 


X Rank 



Ml 


3.89 


2 


4.00 


4 


3.33 


8 


M2 


2.44 


8 


3.46 


7* 


4. 00 


4* 


M3 


3.46 


5 


3.54 


6 


3.67 


7 


M4 


3.96 


1 


4.15 


3 


4. 33 


2* 


M5 


3.74 


4 


3.46 


7* 


4.00 


4* 


M6 


3.23 


7 


4. 25 


2 


5.00 


1 


M7 


3.41 


6 


3.62 


5 


4.00 


4* 


M8 


3.77 


3 


4. 31 


1 


4. 33 


2* 



3.49 3. 35 4.08 



Indicates tied ranking 



0 

ERIC 



- 80 - 



87 



GROUP N 



Objective 
(With Sample 
Test Item) 


Students 
n = 38 
X Rank 


Parents 
n = 15 
X Rank 


Teachers 
n = 4 
X Rank 


Nl 


3.10 


5 


3.80 




5 


4.00 


4* 


N2 


3.27 


3 


4.01 




3 


4.00 


4 * 


N3 


3.05 


6 


3.67 




7 


4. 50 


1 * 


N4 


3.53 


1 


4. 33 




1 


3.50 


7 


N5 


2.70 


8 


3.73 




6 


4.00 


4 * 


N 6 


3.46 


2 


4. 20 




2 


4.50 


1 * 


N7 


2.79 


7 


3.60 




8 


3.25 


8 


N 8 


3.19 


4 


3.86 




4 


4.25 


3 




3.14 




3.90 






4. CO 




Objective 


Students 


Parents 




Teachers 


fithout Sample 


n = 


28 


n = 


16 




n = 


3 


Test Item) 


X 


Rank 


X 


Rank 


X 


Rank 



Nl 


3.18 


1 


N2 


3.04 


3* 


N3 


2.39 


8 


N4 


3.00 


5 


N5 


2.67 


7 


N 6 


3.14 


2 


N7 


3.04 


3* 


N 8 


2.96 


6 



2.93 



*indicates tied ranking 



3.88 


1 * 


4.33 


2 * 


3.87 


3 


4. 33 


2 * 


3.38 


8 


2.67 


8 


3.63 


6 


3.67 


5* 


3.40 


7 


3.67 


5* 


3.88 


1 * 


4. 33 


2 * 


3.81 


4 


3.67 


5* 


3.69 


5 


4.67 


1 



3.69 3.92 
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GROUP P 



Objective 


Students 


Parents 


Teachers 


(With Sample 


n = 43 


n = 17 


n = 4 


Test Item) 


X Rank 


X Rank 


X Rank 



PI 


2.69 


9 


2.88 


9 


3.50 


8 


P2 


4. 30 


4 


4.18 


6 


4.50 


1* 


P3 


3.74 


1 


4. 71 


2 


4.50 


1* 


P4 


3.14 


7* 


3.41 


8 


3.25 


9 


P5 


3.57 


3 


4.63 


3 


4. 25 


4 


P6 


3.65 


2 


4. 76 


1 


4.50 


1* 


P7 


3.30 


5 


4.47 


4 


4.00 


5* 


P8 


3.14 


7* 


3.65 


7 


4.00 


5* 


F9 


3.17 


6 


4. 24 


5 


4.00 


5* 




3.41 




4. 10 




4.06 




Objective 


Students 


Parents 


Teachers 


(Without Sample 


n = 


24 


n = 


13 


n = 


3 


Test Item 


X 


Rank 




Rank 


X 


Rank 



PI 


2.00 


9 


3.00 


9 


2.67 


9 


P2 


3.04 


5 


3.85 


5 


3.00 


7* 


P3 


3.79 


1 


4.46 


1 


4.67 


1* 


P4 


2.82 


6 


3.31 


8 


3.00 


7* 


P5 


2.79 


7 


3.75 


6 


3.67 


4* 


P6 


3.75 


2* 


4.15 


3 


3.67 


4* 


P7 


2.78 


8 


3.91 


4 


4.67 


1* 


P8 


3.75 


2* 


3.54 


7 


4. CO 


6 


P9 


3.63 


4 


4.23 


2 


4.67 


1* 




3. 15 




3.80 




3.78 





*indicates tied ranking 
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Table 12. 



Rank Order Intercorrelations Among Objectives by 
Students, Parents, and Teachers (7-12 Grade Levels). 



Objective Set M 

With sample test items 
Without sample test items 



Student/ 

Parent 



.37 
. 39 



Student/ 

Teacher 



.08 

-.11 



Parent/ 

Teacher 



-.13 

.54 



Objective Set N 

With sample test items .93* .15 

Without sample test items .96* .53 



.14 

. 75* 



Objective Set P 

With sample test items .93* 

Without sample test items . 69* 



. 88 * . 75 * 

. 39 . 84* 



* significant at p< .05 level 



by subjects relating rankings of objectives with sample test 
items to those without sample test items. In other words, 
parent rankings of objectives without sample test items are 
correlated with the parent rankings of objectives with sample 
test items u 

As can be seen, only eight of the 33 correlations were sig- 
nificant at the p< .05 level. There are probably two major 
reasons why significant correlations were net obtained. First, 
if the range within which all the mean ratings are operating 
is very small, then even small differences due to chance vari- 
abrlity could cause ranking changes and this, in turn, could 
Y^ld low correlations. Looking at the means presented in 
Tables 7, 8, and 9 shows that 80 per cent of all these means 
fall within the 3.50 to 4.50 range. This suggests that even 
small variations due to chance could change the relative rank- 
ing of an objective enough to cause non-significant correla- 
tions. As an example, we can examine mean ratings by students 
for the D set of objectives: 
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% 



Mean Ratings and Rankings ( in parentheses ) 



Objective With sample Without sample Difference 
test item test item 



Dl 


3.57 


D2 


3.79 


D3 


3.65 


D4 


3.60 


D5 


3.72 


D6 


3.38 


D7 


3.39 


D8 


3.39 



(5) 


3.14 


(1) 


3.74 


(3) 


3.46 


(4) 


3.52 


(2) 


3.24 


(8) 


4.04 


(6) 


3.00 


(6) 


3.33 



(7) 


.43 


(2) 


.05 


(4) 


.21 


(3) 


.08 


(6) 


.48 


(1) 


.66 


(8) 


.39 


(5) 


.06 



The difference for any single objective appears to be rela- 
tively small, but as can be seen by looking at Table 13, the 
rank order correlation for this particular set of data was only 
.09. Looking at the means and their difference for Objec- 
tive D6 reveals that a difference of . 66 on a five point scale 
caused this particular obj active to be the lowest ranked ob- 
jective when coupled with a sample test item and the highest 
ranked objective when the sample test item was omitted. Again, 
this may be explained by the very small range (3.00 to 4.04) 
within which all the means fell. 



Table 13. Rank Order Correlations Among Objectives With and 
Without Sample Test Items by Students, Parents, 
and Teachers. 



Objective Set Students Parents 



Teachers 



A 




.47 


.70* 


B 




.35 


.27 


C 




.54 


.15 


D 


.09 


.17 


.13 


E 


.78* 


.78* 


.81* 


F 


.10 


.73* 


. 26 


G 


.22 


. 22 


.80* 


H 


.58 


.47 


. 33 


J 


.01 


.04 


.89* 


M 


.50 


.64 


.28 


N 


. 36 


.33 


. 24 


P 


.52 


.80* 


. 36 



^significant at p< .05 level 
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A second major factor contributing to the non-significant 
correlations found in Table 13 seems to have been the sample 
test item itself. Looking at the differences of mean ratings 
■with and without sample test items revealed some gross dif- 
ferences. If we take the mean rating for an objective with 
sample test item and subtract the mean rating for the same 
objective without the sample test item, then perform this 
operation for every objective in the survey, we get a range 
1.60 to —.85 (where a minus difference indicates that the 
mean rating for the objective without sample test item was 
l^cfhex than the mean rating for the same objective with a 
sample test item.) The large differences, which appear much 
greater than those which chance would introduce, are best ex- 
plained by looking at the content of the objective and/or the 
sample test item. 

Including a sample test item with the objective to be rated 
can have one of two effects, if it causes any effect at all. 

It can cause the mean rating of the objective to be increased 
decreased. It can possibly increase the mean rating by 
c d- ari fying an objective which otherwise would be unclear. It 
can possibly decrease the mean rating by introducing complica- 
tions, such as very technical language, which otherwise are 
not in the objective. 

Assuming that large differences in mean ratings between ob- 
jectives with and without sample test items are caused by 
the items themselves, a separate content analysis was com- 
pleted. Those seven objectives which had the largest positive 
differences in means (meaning the objective with sample test 
item) as rated by parents and students were randomly mixed 

the objectives and have the six largest negative differ- 
ences (the objective without sample test item had the higher 
rating). Members of the project staff were then asked to 
determine which way the objective would receive the higher 
rating, with or without sample test item, and why. 

ihe results of this analysis proved to be inconclusive. In 

of the cases was the staff able to unanimously pre— 

. lc ‘t in 'which form the objective would receive the higher rat— 
five of the remaining ten cases, 50 per cent or more 
°f the staff were unable to correctly predict which form 
would receive the higher rating. 

An examination of the objectives and the sample test items 
failed to yield any obvious differences between those which 
were correctly predicted and those which were not. 

Looking at all the sets of objectives A through P, we find 
in every case where there is a statistically significant dif- 
ference in the average mean rating for the entire set of ob- 
jectives by rating group (students or parents), the difference 
was positive, that is, the objective set with sample test 
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items had a higher average rating than did the set without 
sample test items. This may indicate that the inclusion of 
sample test items tends to cause higher ratings. 

A part of the field trial design involved using classes with 
differing socioeconomic backgrounds . Objective sets D, E, 

F, G, H, and J were all formed by random selection from the 
same IOX objective collection (the Study Skills section of 
the 4—6 Reading Collection). Objective sets D, E, and F were 
distributed to classrooms which were composed of students 
mainly from middle and upper class English speaking back- 
grounds. Sets G, H, and J were distributed to classrooms 
composed mainly of students with Spanish speaking backgrounds. 

For the student populations which rated G, H, and J, the dif- 
ferences in the means for each objective with and without 
sample test items were, with one exception, positive. These 
positive differences were high; 19 of the 26 means differed 
by .50 and above. In sets D, E, and F, the differences were 
in both the positive and negative directions, but only five 
of the 26 had an absolute value of .50 or greater.. This may 
indicate that the sample test item has a greater positive ef- 
fect (e.g., may clarify the objective) when the population 
rating the objective has a language background differing from 
that used in the objective. 



Controlling the Quality of Test Items 

In addition to the more general task of evaluating the quality 
of the instructional objectives developed during the project, 
there was the problem of developing measuring procedures 
which adequately assessed these objectives. Two lines of at- 
tack on this problem were employed during the project. First, 
we employed an a priori strategy including the production of 
careful specifications for item writers. Second, several a 
posteriori approaches were tested that involved the use of 
tryout data secured after the items had been developed and 
administered to appropriate learners. The final two sections 
of the chapter will examine both of these lines of endeavor. 





